Nov 03 19:54:08.939139 ip-10-0-132-25 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 19:54:08.939150 ip-10-0-132-25 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 19:54:08.939156 ip-10-0-132-25 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 19:54:08.939373 ip-10-0-132-25 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 19:54:19.188917 ip-10-0-132-25 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 19:54:19.188934 ip-10-0-132-25 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 92a6cca9cfaa45bc9cabad785243a6ec -- Nov 03 19:56:21.761667 ip-10-0-132-25 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:56:22.140828 ip-10-0-132-25 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:56:22.140828 ip-10-0-132-25 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:56:22.140828 ip-10-0-132-25 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:56:22.140828 ip-10-0-132-25 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:56:22.144536 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144460 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:22.144536 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144537 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144541 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144544 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144548 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144560 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144563 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144566 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144569 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144571 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144574 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144577 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144579 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144582 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144584 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144587 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144590 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144592 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144595 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144598 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144600 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:22.144609 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144603 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144606 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144608 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144611 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144614 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144617 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144620 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144622 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144626 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144628 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144631 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144633 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144636 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144639 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144641 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144644 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144646 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144655 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144657 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144660 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:22.145090 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144662 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144665 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144667 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144671 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144673 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144676 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144679 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144681 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144683 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144686 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144688 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144691 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144693 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144696 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144700 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144703 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144705 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144708 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144711 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144713 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:22.145577 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144716 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144718 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144721 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144724 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144726 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144729 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144731 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144734 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144737 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144739 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144747 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144750 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144753 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144756 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144758 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144760 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144763 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144765 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144768 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:22.146047 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144773 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144777 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144780 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144784 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144787 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144789 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144792 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144796 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144798 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144801 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144803 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144806 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144808 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144811 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144813 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144816 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144818 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144821 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144824 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144826 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:22.146627 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144829 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:22.147102 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144831 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:22.147102 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144834 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:22.147102 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144836 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:22.147102 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144850 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:22.147102 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.144853 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147102 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147111 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147114 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147118 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147121 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147124 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147127 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147129 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147132 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147136 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147140 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147143 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147146 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147149 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147151 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147154 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147157 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147160 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:22.147225 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147163 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147165 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147168 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147171 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147174 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147176 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147179 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147181 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147184 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147186 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147189 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147191 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147194 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147199 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147203 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147207 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147210 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147213 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147216 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147218 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:22.147710 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147221 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147224 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147227 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147229 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147233 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147235 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147238 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147241 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147243 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147247 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147251 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147254 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147256 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147259 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147261 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147264 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147267 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147270 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147272 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147274 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:22.148196 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147277 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147280 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147282 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147285 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147288 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147290 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147293 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147296 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147299 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147302 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147305 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147307 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147310 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147312 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147315 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147318 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147321 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147324 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147326 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147329 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:22.148691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147331 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147334 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147336 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147339 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147342 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147344 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147347 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147349 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147352 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147354 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147357 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147359 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147362 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147365 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147367 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147370 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147373 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147375 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147378 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:22.149154 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147381 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147399 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147402 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147405 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147408 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147410 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147413 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147416 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147419 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147898 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147903 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147906 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147909 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147912 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147915 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147918 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147921 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147924 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147927 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:22.149626 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147929 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147933 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147937 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147941 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147944 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147948 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147951 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147953 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147956 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147958 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147961 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147964 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147966 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147970 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147972 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147975 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147978 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147981 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147983 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:22.150070 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147986 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147988 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147991 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147993 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147996 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.147998 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148001 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148003 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148006 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148009 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148011 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148014 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148017 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148019 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148022 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148025 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148027 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148030 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148032 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148035 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:22.150559 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148037 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148040 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148042 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148045 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148047 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148050 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148053 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148056 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148058 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148061 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148063 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148066 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148068 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148071 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148073 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148076 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148078 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148080 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148083 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148085 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:22.151032 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148088 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148090 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148093 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148095 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148098 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148100 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148103 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148106 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148110 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148113 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148117 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148120 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148132 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148135 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148138 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148140 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148143 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148146 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148148 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:22.151529 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148151 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148155 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148157 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148160 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148163 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148165 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148168 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148170 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148173 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148175 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148178 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148180 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148183 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148187 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148190 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148193 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148196 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.148198 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149093 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149103 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149110 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:56:22.152008 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149115 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149120 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149123 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149128 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149134 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149140 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149145 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149149 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149153 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149157 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149161 2579 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149164 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149167 2579 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149171 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149175 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149179 2579 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149182 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149185 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149189 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149192 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149196 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149199 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149202 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:56:22.152521 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149205 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149208 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149213 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149218 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149224 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149227 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149232 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149235 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149238 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149240 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149244 2579 flags.go:64] FLAG: --enable-server="true" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149247 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149251 2579 flags.go:64] FLAG: --event-burst="100" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149254 2579 flags.go:64] FLAG: --event-qps="50" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149256 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149259 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149263 2579 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149267 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149270 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149273 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149276 2579 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149279 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149286 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149291 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149295 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:56:22.153090 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149300 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149322 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149326 2579 flags.go:64] FLAG: --feature-gates="" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149330 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149334 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149337 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149340 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149344 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149347 2579 flags.go:64] FLAG: --help="false" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149351 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149354 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149362 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149365 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149369 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149374 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149379 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149394 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149398 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149401 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149404 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149407 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149410 2579 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149413 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149416 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:56:22.153770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149419 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149422 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149425 2579 flags.go:64] FLAG: --lock-file="" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149428 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149431 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149436 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149443 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149446 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149451 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149455 2579 flags.go:64] FLAG: --logging-format="text" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149460 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149465 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149470 2579 flags.go:64] FLAG: --manifest-url="" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149474 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149480 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149483 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149493 2579 flags.go:64] FLAG: --max-pods="110" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149496 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149499 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149504 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149507 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149511 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149514 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149517 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149524 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:56:22.154351 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149528 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149531 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149534 2579 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149537 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149542 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149548 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149553 2579 flags.go:64] FLAG: --port="10250" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149556 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149559 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-08ee6972ad3dc691c" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149563 2579 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149566 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149569 2579 flags.go:64] FLAG: --register-node="true" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149573 2579 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149577 2579 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149580 2579 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149584 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149587 2579 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149591 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149594 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149597 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149600 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149603 2579 flags.go:64] FLAG: --runonce="false" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149606 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149610 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149613 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149616 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149622 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:56:22.155033 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149627 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149632 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149637 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149640 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149643 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149646 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149649 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149652 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149655 2579 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149658 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149663 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149666 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149669 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149674 2579 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149677 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149680 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149682 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149687 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149690 2579 flags.go:64] FLAG: --v="2" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149694 2579 flags.go:64] FLAG: --version="false" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149701 2579 flags.go:64] FLAG: --vmodule="" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149709 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.149714 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149857 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:22.155707 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149864 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149870 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149873 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149876 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149879 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149881 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149884 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149888 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149891 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149893 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149896 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149899 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149901 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149904 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149907 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149909 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149912 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149915 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149917 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:22.156615 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149920 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149922 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149925 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149927 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149930 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149935 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149941 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149945 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149949 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149951 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149954 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149957 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149960 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149963 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149965 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149968 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149970 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149973 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149976 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:22.157201 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149978 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149982 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149985 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149988 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149991 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149993 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149996 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.149999 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150001 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150004 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150006 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150009 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150014 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150018 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150022 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150027 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150030 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150032 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150034 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150038 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:22.157691 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150041 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150044 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150047 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150049 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150052 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150056 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150058 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150061 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150064 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150066 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150070 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150074 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150077 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150081 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150084 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150087 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150090 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150094 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150099 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150103 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:22.158168 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150106 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150109 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150111 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150114 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150116 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150119 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150122 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150124 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150127 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150129 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150132 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150136 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150138 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150142 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150146 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150149 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150151 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150154 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150164 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150167 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:22.158663 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150171 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150175 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150180 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150184 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150187 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150191 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150194 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.150221 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150271 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150276 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150279 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150282 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150285 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150287 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150290 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150293 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:22.159133 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150295 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150298 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150301 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150304 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150306 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150309 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150312 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150314 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150317 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150319 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150322 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150325 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150327 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150331 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150335 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150340 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150344 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150348 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150350 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:22.159580 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150353 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150355 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150359 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150362 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150365 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150367 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150370 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150372 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150375 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150378 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150380 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150394 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150397 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150400 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150402 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150405 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150408 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150411 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150415 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150420 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:22.160033 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150424 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150428 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150431 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150434 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150436 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150439 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150441 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150444 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150446 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150450 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150452 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150456 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150460 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150462 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150466 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150469 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150471 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150474 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150476 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150479 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:22.160542 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150482 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150484 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150487 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150491 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150495 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150499 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150503 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150509 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150512 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150515 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150517 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150520 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150523 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150526 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150530 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150534 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150537 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150540 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150543 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150546 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:22.161021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150548 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150551 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150554 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150557 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150559 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150562 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150566 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150571 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150575 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150579 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150584 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150586 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150589 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150592 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150594 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150596 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150599 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150604 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.150606 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:22.161499 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.158259 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.158276 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.158835 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.158847 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158908 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158913 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158916 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158919 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158922 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158925 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158928 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158930 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158933 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158937 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158940 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158942 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158945 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158947 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158950 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:22.161938 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158953 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158956 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158958 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158961 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158964 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158966 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158969 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158971 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158975 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158978 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158980 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158983 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158986 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158989 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158992 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158995 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.158998 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159000 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159003 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:22.162436 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159006 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159008 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159012 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159014 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159017 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159019 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159022 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159024 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159027 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159030 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159032 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159035 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159038 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159041 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159045 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159049 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159053 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159056 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159059 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:22.162885 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159061 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159064 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159067 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159070 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159074 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159077 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159080 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159083 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159086 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159089 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159092 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159095 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159098 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159100 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159103 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159106 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159109 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159111 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159114 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:22.163320 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159116 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159119 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159122 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159124 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159127 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159130 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159132 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159136 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159138 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159141 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159144 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159147 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159150 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159152 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159155 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159157 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159160 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159163 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159165 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159169 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:22.163787 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159171 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159174 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159177 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159180 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159182 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159185 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159188 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159190 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159193 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159196 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159198 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159201 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159204 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159206 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.159234 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159373 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:22.164259 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159379 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159382 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159398 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159404 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159408 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159411 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159414 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159417 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159420 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159422 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159425 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159427 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159430 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159433 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159436 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159438 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159441 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159444 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159447 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:22.164683 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159450 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159452 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159455 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159457 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159461 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159465 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159469 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159472 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159475 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159477 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159480 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159483 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159485 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159488 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159491 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159493 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159496 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159498 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159502 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:22.165132 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159505 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159507 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159510 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159512 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159515 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159517 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159520 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159522 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159525 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159528 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159530 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159533 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159536 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159538 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159541 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159544 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159546 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159549 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159552 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159555 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:22.165595 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159557 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159560 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159563 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159565 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159568 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159571 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159574 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159576 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159579 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159582 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159584 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159587 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159591 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159593 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159596 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159599 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159601 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159604 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159606 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:22.166119 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159609 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159611 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159614 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159617 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159620 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159622 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159625 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159631 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159635 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159638 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159640 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159643 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159646 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159649 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159652 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159655 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159658 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159660 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159663 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:22.166593 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159666 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159668 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159671 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159673 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159676 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159678 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159681 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159684 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159687 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.159710 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159743 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159747 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159750 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159753 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159757 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159760 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:56:22.167040 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159763 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159766 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159769 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159771 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159774 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159776 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159779 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159782 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159784 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159787 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159789 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159792 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159795 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159797 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159800 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159803 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159807 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159811 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159814 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:56:22.167567 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159817 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159820 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159823 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159826 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159829 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159832 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159834 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159837 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159839 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159842 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159845 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159847 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159849 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159852 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159854 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159857 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159860 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159862 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159865 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159868 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:56:22.168021 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159870 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159873 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159875 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159878 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159880 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159883 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159886 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159889 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159891 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159894 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159896 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159899 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159902 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159904 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159906 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159909 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159912 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159914 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159917 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:56:22.168513 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159920 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159922 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159925 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159929 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159933 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159936 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159939 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159942 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159944 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159947 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159950 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159952 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159955 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159957 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159960 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159962 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159965 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159967 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159970 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:56:22.168946 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159972 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159975 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159977 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159980 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159982 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159988 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159991 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159993 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159996 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.159999 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.160001 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.160004 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.160006 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.160010 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.160013 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.160016 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.160018 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.160021 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.160024 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.160026 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:56:22.169456 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.160029 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:56:22.169906 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.160032 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:56:22.169906 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:22.160035 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:56:22.169906 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.160086 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:56:22.169906 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.163304 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 19:56:22.169906 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.164241 2579 server.go:1014] "Starting client certificate rotation" Nov 03 19:56:22.169906 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.164338 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:56:22.169906 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.164372 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:56:22.169906 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.166079 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:56:22.170184 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.170167 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:56:22.177497 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.177481 2579 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:56:22.181052 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.181039 2579 log.go:25] "Validated CRI v1 image API" Nov 03 19:56:22.181913 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.181886 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:56:22.185081 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.185062 2579 fs.go:135] Filesystem UUIDs: map[581dc9db-1ea1-4a6b-a1a5-5aee7ed5d764:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f0a35fe7-61fb-4ae0-bb00-00e3feabca72:/dev/nvme0n1p4] Nov 03 19:56:22.185140 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.185079 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 19:56:22.190102 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.189990 2579 manager.go:217] Machine: {Timestamp:2026-11-03 19:56:22.188686858 +0000 UTC m=+0.316415927 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101968 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f64289fc145b566f89e0965145e68 SystemUUID:ec2f6428-9fc1-45b5-66f8-9e0965145e68 BootID:92a6cca9-cfaa-45bc-9cab-ad785243a6ec Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d0:63:95:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d0:63:95:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:9d:a2:e8:f9:20 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:56:22.190499 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.190488 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:56:22.190587 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.190575 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:56:22.193857 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.193809 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:56:22.194180 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.193863 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-25.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:56:22.194746 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.194718 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:56:22.194746 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.194748 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:56:22.194884 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.194827 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:56:22.195054 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.195037 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:56:22.195814 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.195803 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:56:22.197459 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.197442 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:56:22.199543 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.199531 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:56:22.199668 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.199659 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:56:22.199709 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.199691 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:56:22.199709 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.199701 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:56:22.199771 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.199711 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:56:22.199771 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.199719 2579 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:56:22.199771 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.199743 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:56:22.200071 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.200060 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:56:22.202280 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.202264 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:56:22.207289 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.207277 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:56:22.208010 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.207998 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:56:22.208055 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.208028 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:56:22.208190 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.208180 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:56:22.208224 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.208196 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:56:22.208224 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.208202 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:56:22.208224 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.208209 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:56:22.208224 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.208215 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:56:22.208224 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.208222 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:56:22.208779 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.208770 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:56:22.208816 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.208781 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:56:22.208816 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.208790 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:56:22.208816 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.208796 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:56:22.208816 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.208804 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:56:22.208816 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.208813 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:56:22.209358 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.209349 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:56:22.209407 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.209358 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:56:22.212974 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.212962 2579 server.go:1263] "Started kubelet" Nov 03 19:56:22.213081 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.213019 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:56:22.213132 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.213064 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:56:22.213194 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.213133 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:56:22.213840 ip-10-0-132-25 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:56:22.214358 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.214338 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:56:22.214535 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.214523 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:56:22.219567 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.218504 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-25.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 19:56:22.219567 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.218511 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-25.us-east-2.compute.internal.18e4a130b7b58dd5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-25.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-25.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:56:22.212931029 +0000 UTC m=+0.340660100,LastTimestamp:2026-11-03 19:56:22.212931029 +0000 UTC m=+0.340660100,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-25.us-east-2.compute.internal,}" Nov 03 19:56:22.220302 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.220279 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:56:22.220379 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.220344 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kcp9g" Nov 03 19:56:22.220828 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.220810 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:56:22.220925 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.220904 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-25.us-east-2.compute.internal\" not found" Nov 03 19:56:22.221886 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.221867 2579 factory.go:55] Registering systemd factory Nov 03 19:56:22.221886 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.221884 2579 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:56:22.222052 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.221959 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 19:56:22.222149 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.222132 2579 factory.go:153] Registering CRI-O factory Nov 03 19:56:22.222207 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.222153 2579 factory.go:223] Registration of the crio container factory successfully Nov 03 19:56:22.222263 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.222218 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:56:22.222263 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.222237 2579 factory.go:103] Registering Raw factory Nov 03 19:56:22.222263 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.222237 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:56:22.222263 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.222239 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:56:22.222263 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.222259 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:56:22.222514 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.222251 2579 manager.go:1199] Started watching for new ooms in manager Nov 03 19:56:22.222514 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.222508 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:56:22.222514 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.222514 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:56:22.223093 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.223077 2579 manager.go:319] Starting recovery of all containers Nov 03 19:56:22.223297 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.223263 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-25.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 19:56:22.223422 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.223360 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 19:56:22.223513 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.223492 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-25.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 19:56:22.231170 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.231145 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 19:56:22.231908 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.231870 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:56:22.232921 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.232892 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kcp9g" Nov 03 19:56:22.234616 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.234597 2579 manager.go:324] Recovery completed Nov 03 19:56:22.239328 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.239310 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:22.241223 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.241202 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:22.241282 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.241236 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:22.241282 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.241251 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:22.241852 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.241840 2579 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:56:22.241852 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.241850 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:56:22.241985 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.241856 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:22.241985 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.241863 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:56:22.242624 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.242605 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:22.242689 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.242637 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:22.242689 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.242651 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:22.243488 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.243422 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-25.us-east-2.compute.internal.18e4a130b9653449 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-25.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-25.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-25.us-east-2.compute.internal,},FirstTimestamp:2026-11-03 19:56:22.241219657 +0000 UTC m=+0.368948729,LastTimestamp:2026-11-03 19:56:22.241219657 +0000 UTC m=+0.368948729,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-25.us-east-2.compute.internal,}" Nov 03 19:56:22.243895 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.243883 2579 policy_none.go:50] "Start" Nov 03 19:56:22.243942 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.243901 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:56:22.243942 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.243911 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:56:22.246014 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.246000 2579 policy_none.go:44] "Start" Nov 03 19:56:22.283042 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.283015 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:56:22.288549 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.288531 2579 manager.go:341] "Starting Device Plugin manager" Nov 03 19:56:22.288969 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.288948 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:56:22.289026 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.288972 2579 server.go:87] "Starting device plugin registration server" Nov 03 19:56:22.289418 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.289238 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:56:22.289467 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.289426 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:56:22.289542 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.289526 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:56:22.289612 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.289599 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:56:22.289612 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.289611 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:56:22.290163 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.290143 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:56:22.290241 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.290187 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-25.us-east-2.compute.internal\" not found" Nov 03 19:56:22.353614 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.353590 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:56:22.353745 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.353620 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:56:22.353745 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.353642 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:56:22.353745 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.353678 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 19:56:22.357496 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.357479 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:22.390488 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.390473 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:22.391343 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.391304 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:22.391343 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.391332 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:22.391454 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.391345 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:22.391454 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.391371 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.400216 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.400198 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.400273 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.400219 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-25.us-east-2.compute.internal\": node \"ip-10-0-132-25.us-east-2.compute.internal\" not found" Nov 03 19:56:22.417252 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.417230 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-25.us-east-2.compute.internal\" not found" Nov 03 19:56:22.454543 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.454505 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal"] Nov 03 19:56:22.454677 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.454571 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:22.455417 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.455381 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:22.455506 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.455428 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:22.455506 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.455441 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:22.457484 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.457472 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:22.457624 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.457609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.457690 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.457641 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:22.458117 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.458102 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:22.458117 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.458109 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:22.458206 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.458132 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:22.458206 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.458146 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:22.458206 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.458135 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:22.458294 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.458226 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:22.460321 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.460308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.460367 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.460333 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:56:22.460997 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.460982 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:56:22.461065 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.461009 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:56:22.461065 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.461020 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:56:22.493286 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.493265 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-25.us-east-2.compute.internal\" not found" node="ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.500163 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.500144 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-25.us-east-2.compute.internal\" not found" node="ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.517750 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.517730 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-25.us-east-2.compute.internal\" not found" Nov 03 19:56:22.524489 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.524471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca7a77d780aa380ba43b79db4d32277a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"ca7a77d780aa380ba43b79db4d32277a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.524589 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.524501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0616d8478d645504ab20b6a243f25dbb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"0616d8478d645504ab20b6a243f25dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.524589 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.524559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0616d8478d645504ab20b6a243f25dbb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"0616d8478d645504ab20b6a243f25dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.618545 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.618518 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-25.us-east-2.compute.internal\" not found" Nov 03 19:56:22.624918 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.624898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca7a77d780aa380ba43b79db4d32277a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"ca7a77d780aa380ba43b79db4d32277a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.624978 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.624927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0616d8478d645504ab20b6a243f25dbb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"0616d8478d645504ab20b6a243f25dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.624978 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.624945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0616d8478d645504ab20b6a243f25dbb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"0616d8478d645504ab20b6a243f25dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.625056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.625009 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca7a77d780aa380ba43b79db4d32277a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"ca7a77d780aa380ba43b79db4d32277a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.625109 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.625083 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0616d8478d645504ab20b6a243f25dbb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"0616d8478d645504ab20b6a243f25dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.625149 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.625130 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0616d8478d645504ab20b6a243f25dbb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"0616d8478d645504ab20b6a243f25dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.719348 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.719290 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-25.us-east-2.compute.internal\" not found" Nov 03 19:56:22.796875 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.796841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.803328 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:22.803047 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:22.819651 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.819631 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-25.us-east-2.compute.internal\" not found" Nov 03 19:56:22.920257 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:22.920224 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-25.us-east-2.compute.internal\" not found" Nov 03 19:56:23.020770 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.020712 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-25.us-east-2.compute.internal\" not found" Nov 03 19:56:23.071977 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.071951 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:23.121123 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.121097 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:23.135067 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.135046 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:56:23.136773 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.136757 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:56:23.145887 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.145868 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:56:23.164024 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.164005 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 19:56:23.164170 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.164151 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 19:56:23.164211 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.164150 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 19:56:23.200247 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.200222 2579 apiserver.go:52] "Watching apiserver" Nov 03 19:56:23.219528 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.219509 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:56:23.219802 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.219786 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-xx4kz","openshift-network-diagnostics/network-check-target-h4zrs","kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal","openshift-multus/multus-5p4wp","openshift-multus/network-metrics-daemon-st6tw","openshift-network-operator/iptables-alerter-zqftx","kube-system/konnectivity-agent-9ktt5","openshift-cluster-node-tuning-operator/tuned-q5hct","openshift-image-registry/node-ca-cwtc9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj","openshift-dns/node-resolver-ssj5l","openshift-ovn-kubernetes/ovnkube-node-sc9nt","kube-system/global-pull-secret-syncer-l5wn9"] Nov 03 19:56:23.220452 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.220436 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 19:56:23.222420 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.222402 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.224486 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.224463 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:23.224574 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.224537 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:23.226400 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.226371 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:56:23.226997 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.226980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.227350 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.227333 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:56:23.227350 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.227342 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:56:23.227498 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.227355 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:56:23.227498 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.227457 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:56:23.227569 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.227517 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l9cjs\"" Nov 03 19:56:23.228593 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.228572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.228646 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.228603 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jw95\" (UniqueName: \"kubernetes.io/projected/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-kube-api-access-7jw95\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.228688 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.228668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-system-cni-dir\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.228721 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.228699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cnibin\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.228721 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.228715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cni-binary-copy\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.228783 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.228757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.228814 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.228781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.228861 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.228844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75qtn\" (UniqueName: \"kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn\") pod \"network-check-target-h4zrs\" (UID: \"c98e82ab-c3b3-4d18-b381-083fdb90b448\") " pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:23.229208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.229196 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:23.229274 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.229257 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:23.231794 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.231776 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:56:23.234010 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.233994 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:56:23.235024 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.235006 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 19:56:23.236317 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.236302 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.238847 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.238829 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:56:23.238847 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.238801 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:51:22 +0000 UTC" deadline="2028-06-27 01:00:36.544502346 +0000 UTC" Nov 03 19:56:23.239000 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.238853 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14429h4m13.305652658s" Nov 03 19:56:23.241602 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.241588 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.244301 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.244284 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:56:23.248434 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.247571 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.250460 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.250449 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:23.250517 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.250503 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:23.252902 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.252888 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8zmh4" Nov 03 19:56:23.263580 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.263562 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8zmh4" Nov 03 19:56:23.320198 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.320175 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:56:23.320198 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.320194 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:23.320423 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.320230 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:56:23.320423 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.320242 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:56:23.320423 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.320246 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:23.320561 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.320546 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:56:23.320632 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.320616 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:56:23.320892 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.320868 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:56:23.320892 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.320868 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:56:23.321495 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.321478 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n8st4\"" Nov 03 19:56:23.321495 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.321478 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:56:23.321607 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.321516 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:56:23.321607 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.321549 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:56:23.321607 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.321561 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:56:23.321607 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.321564 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gc46m\"" Nov 03 19:56:23.321607 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.321605 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:56:23.321761 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.321691 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:56:23.321761 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.321721 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:56:23.321949 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.321935 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:56:23.322007 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.321991 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5vrk4\"" Nov 03 19:56:23.322683 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.322672 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:56:23.322756 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.322706 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:56:23.322756 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.322739 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:56:23.322866 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.322856 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6tnlf\"" Nov 03 19:56:23.322939 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.322920 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:56:23.323000 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.322922 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4n975\"" Nov 03 19:56:23.323000 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.322951 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lp2l4\"" Nov 03 19:56:23.323000 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.322932 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wsxsf\"" Nov 03 19:56:23.323106 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.323095 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p44lh\"" Nov 03 19:56:23.323160 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.323149 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:56:23.323304 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.323293 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:56:23.329915 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.329899 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.329993 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.329924 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-cni-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.329993 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.329945 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/642aaac2-4dc2-4800-bc55-c70730884400-konnectivity-ca\") pod \"konnectivity-agent-9ktt5\" (UID: \"642aaac2-4dc2-4800-bc55-c70730884400\") " pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:56:23.329993 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.329961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysctl-conf\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.330115 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-socket-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.330115 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.330115 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330064 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbht\" (UniqueName: \"kubernetes.io/projected/7f215e51-f5cf-4103-9904-693d5721e647-kube-api-access-mgbht\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:56:23.330212 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-env-overrides\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.330212 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cni-binary-copy\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.330273 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-netns\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.330309 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysconfig\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.330347 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330309 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.330347 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.330437 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330396 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.330437 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovn-node-metrics-cert\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.330518 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330490 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-system-cni-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.330565 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330515 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-etc-kubernetes\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.330565 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn92t\" (UniqueName: \"kubernetes.io/projected/33806d61-5ac2-49ba-9fc9-c89046f2ff03-kube-api-access-tn92t\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:23.330641 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-sys-fs\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.330641 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330597 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cni-binary-copy\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.330727 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-log-socket\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.330727 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:23.330727 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-system-cni-dir\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qtn\" (UniqueName: \"kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn\") pod \"network-check-target-h4zrs\" (UID: \"c98e82ab-c3b3-4d18-b381-083fdb90b448\") " pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffa0c959-6bd1-4feb-b713-3545a6896409-serviceca\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-registration-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f215e51-f5cf-4103-9904-693d5721e647-tmp-dir\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330822 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-system-cni-dir\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330851 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-var-lib-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330884 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-node-log\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-kubelet\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-modprobe-d\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.330956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-run\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-var-lib-kubelet\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-socket-dir-parent\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-slash\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-etc-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.331996 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-cni-bin\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-cnibin\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-k8s-cni-cncf-io\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysctl-d\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffa0c959-6bd1-4feb-b713-3545a6896409-host\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331286 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-sys\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-systemd\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2590e213-dc39-406a-8094-b4e01162e221-cni-binary-copy\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331355 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331411 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b33b71b3-287e-4516-9d0a-726f302a8dff-iptables-alerter-script\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lplg9\" (UniqueName: \"kubernetes.io/projected/308feccf-7178-48c7-b370-3c78f96e8919-kube-api-access-lplg9\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331440 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331452 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-systemd-units\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovnkube-config\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-multus-certs\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.332664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b33b71b3-287e-4516-9d0a-726f302a8dff-host-slash\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331577 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/642aaac2-4dc2-4800-bc55-c70730884400-agent-certs\") pod \"konnectivity-agent-9ktt5\" (UID: \"642aaac2-4dc2-4800-bc55-c70730884400\") " pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-systemd\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-ovn\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovnkube-script-lib\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jw95\" (UniqueName: \"kubernetes.io/projected/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-kube-api-access-7jw95\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331737 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kwj4\" (UniqueName: \"kubernetes.io/projected/2590e213-dc39-406a-8094-b4e01162e221-kube-api-access-8kwj4\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qspj\" (UniqueName: \"kubernetes.io/projected/b33b71b3-287e-4516-9d0a-726f302a8dff-kube-api-access-8qspj\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-host\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-kubelet-config\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331849 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-cni-bin\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2590e213-dc39-406a-8094-b4e01162e221-multus-daemon-config\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/308feccf-7178-48c7-b370-3c78f96e8919-etc-tuned\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331911 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hggp5\" (UniqueName: \"kubernetes.io/projected/ffa0c959-6bd1-4feb-b713-3545a6896409-kube-api-access-hggp5\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-device-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz2q2\" (UniqueName: \"kubernetes.io/projected/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-kube-api-access-wz2q2\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.333056 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.331999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-lib-modules\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/308feccf-7178-48c7-b370-3c78f96e8919-tmp\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332104 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-cni-netd\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cnibin\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-hostroot\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332165 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cnibin\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332192 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp952\" (UniqueName: \"kubernetes.io/projected/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-kube-api-access-fp952\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332215 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-dbus\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-cni-multus\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-kubernetes\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332278 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f215e51-f5cf-4103-9904-693d5721e647-hosts-file\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332319 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-kubelet\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-conf-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332436 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332515 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-run-netns\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.333510 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.332523 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.339762 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.339746 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:23.339841 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.339761 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:23.339841 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.339776 2579 projected.go:196] Error preparing data for projected volume kube-api-access-75qtn for pod openshift-network-diagnostics/network-check-target-h4zrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:23.339938 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.339848 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn podName:c98e82ab-c3b3-4d18-b381-083fdb90b448 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:23.839814446 +0000 UTC m=+1.967543518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-75qtn" (UniqueName: "kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn") pod "network-check-target-h4zrs" (UID: "c98e82ab-c3b3-4d18-b381-083fdb90b448") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:23.340449 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.340414 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:56:23.345459 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.345437 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jw95\" (UniqueName: \"kubernetes.io/projected/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-kube-api-access-7jw95\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.400129 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:23.400102 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca7a77d780aa380ba43b79db4d32277a.slice/crio-35150edf568d96bc082d83b80c1db26cdb6a69c1259c86011542b4cc61f141a6 WatchSource:0}: Error finding container 35150edf568d96bc082d83b80c1db26cdb6a69c1259c86011542b4cc61f141a6: Status 404 returned error can't find the container with id 35150edf568d96bc082d83b80c1db26cdb6a69c1259c86011542b4cc61f141a6 Nov 03 19:56:23.400525 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:23.400506 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0616d8478d645504ab20b6a243f25dbb.slice/crio-bd6773b2f462d346511693cad2335a5ba671730d1886dde1e05ec240083a3169 WatchSource:0}: Error finding container bd6773b2f462d346511693cad2335a5ba671730d1886dde1e05ec240083a3169: Status 404 returned error can't find the container with id bd6773b2f462d346511693cad2335a5ba671730d1886dde1e05ec240083a3169 Nov 03 19:56:23.405071 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.405050 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:56:23.432739 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.432707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kwj4\" (UniqueName: \"kubernetes.io/projected/2590e213-dc39-406a-8094-b4e01162e221-kube-api-access-8kwj4\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.432823 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.432751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qspj\" (UniqueName: \"kubernetes.io/projected/b33b71b3-287e-4516-9d0a-726f302a8dff-kube-api-access-8qspj\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:56:23.432823 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.432769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-host\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.432823 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.432793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-kubelet-config\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:23.432958 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.432823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-cni-bin\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.432958 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.432845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2590e213-dc39-406a-8094-b4e01162e221-multus-daemon-config\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.432958 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.432864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/308feccf-7178-48c7-b370-3c78f96e8919-etc-tuned\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.432958 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.432882 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-host\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.432958 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.432887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hggp5\" (UniqueName: \"kubernetes.io/projected/ffa0c959-6bd1-4feb-b713-3545a6896409-kube-api-access-hggp5\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:56:23.432958 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.432938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-device-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.433208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.432953 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-cni-bin\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.433208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.432965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz2q2\" (UniqueName: \"kubernetes.io/projected/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-kube-api-access-wz2q2\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.433208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-lib-modules\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.433208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433026 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/308feccf-7178-48c7-b370-3c78f96e8919-tmp\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.433208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-cni-netd\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.433208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433071 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-hostroot\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.433208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp952\" (UniqueName: \"kubernetes.io/projected/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-kube-api-access-fp952\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.433208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-dbus\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:23.433208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-cni-multus\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.433208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433156 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-lib-modules\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.433208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-kubernetes\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.433208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433195 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-kubernetes\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.433208 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f215e51-f5cf-4103-9904-693d5721e647-hosts-file\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433235 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-kubelet\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433248 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-device-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-conf-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433292 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-hostroot\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-run-netns\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433331 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-cni-multus\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-cni-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433123 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-cni-netd\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433409 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f215e51-f5cf-4103-9904-693d5721e647-hosts-file\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433446 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-conf-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433463 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433482 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-kubelet\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433493 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-run-netns\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433500 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-cni-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/642aaac2-4dc2-4800-bc55-c70730884400-konnectivity-ca\") pod \"konnectivity-agent-9ktt5\" (UID: \"642aaac2-4dc2-4800-bc55-c70730884400\") " pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433536 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-dbus\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:23.433740 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysctl-conf\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-socket-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433557 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-kubelet-config\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbht\" (UniqueName: \"kubernetes.io/projected/7f215e51-f5cf-4103-9904-693d5721e647-kube-api-access-mgbht\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433648 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433681 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-env-overrides\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-netns\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433732 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysctl-conf\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433752 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-netns\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysconfig\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433795 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433844 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysconfig\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovn-node-metrics-cert\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433879 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-system-cni-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433884 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.434485 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-etc-kubernetes\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433918 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2590e213-dc39-406a-8094-b4e01162e221-multus-daemon-config\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn92t\" (UniqueName: \"kubernetes.io/projected/33806d61-5ac2-49ba-9fc9-c89046f2ff03-kube-api-access-tn92t\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-sys-fs\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.433978 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-socket-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-log-socket\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434011 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-system-cni-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434039 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-sys-fs\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434056 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-log-socket\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffa0c959-6bd1-4feb-b713-3545a6896409-serviceca\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434077 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-etc-kubernetes\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434098 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-registration-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434138 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f215e51-f5cf-4103-9904-693d5721e647-tmp-dir\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-var-lib-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.434168 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-node-log\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.435235 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434174 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-registration-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-kubelet\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434227 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-var-lib-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.434233 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret podName:e6338d09-ef6a-4cf5-8a06-bd466bc5076d nodeName:}" failed. No retries permitted until 2026-11-03 19:56:23.934217867 +0000 UTC m=+2.061946949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret") pod "global-pull-secret-syncer-l5wn9" (UID: "e6338d09-ef6a-4cf5-8a06-bd466bc5076d") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434240 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-kubelet\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434251 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-modprobe-d\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-run\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434276 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-node-log\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434297 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-var-lib-kubelet\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-socket-dir-parent\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-slash\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434357 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-modprobe-d\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-etc-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434410 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-slash\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-cni-bin\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-cnibin\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434427 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-var-lib-kubelet\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.435882 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434454 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-socket-dir-parent\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-k8s-cni-cncf-io\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysctl-d\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434491 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-etc-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434496 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-run\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffa0c959-6bd1-4feb-b713-3545a6896409-host\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434345 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/642aaac2-4dc2-4800-bc55-c70730884400-konnectivity-ca\") pod \"konnectivity-agent-9ktt5\" (UID: \"642aaac2-4dc2-4800-bc55-c70730884400\") " pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434543 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f215e51-f5cf-4103-9904-693d5721e647-tmp-dir\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434552 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-cnibin\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434553 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffa0c959-6bd1-4feb-b713-3545a6896409-serviceca\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434563 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-k8s-cni-cncf-io\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434599 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-cni-bin\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434599 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysctl-d\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434608 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-sys\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434628 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffa0c959-6bd1-4feb-b713-3545a6896409-host\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-systemd\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434646 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-sys\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434681 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2590e213-dc39-406a-8094-b4e01162e221-cni-binary-copy\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.436310 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434697 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b33b71b3-287e-4516-9d0a-726f302a8dff-iptables-alerter-script\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434678 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-systemd\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.434803 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.434841 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs podName:33806d61-5ac2-49ba-9fc9-c89046f2ff03 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:23.934829237 +0000 UTC m=+2.062558304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs") pod "network-metrics-daemon-st6tw" (UID: "33806d61-5ac2-49ba-9fc9-c89046f2ff03") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lplg9\" (UniqueName: \"kubernetes.io/projected/308feccf-7178-48c7-b370-3c78f96e8919-kube-api-access-lplg9\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434889 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-systemd-units\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovnkube-config\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434952 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-systemd-units\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.434965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-multus-certs\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b33b71b3-287e-4516-9d0a-726f302a8dff-host-slash\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/642aaac2-4dc2-4800-bc55-c70730884400-agent-certs\") pod \"konnectivity-agent-9ktt5\" (UID: \"642aaac2-4dc2-4800-bc55-c70730884400\") " pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435085 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-systemd\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435126 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b33b71b3-287e-4516-9d0a-726f302a8dff-host-slash\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435137 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-ovn\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435089 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-multus-certs\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.436796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435159 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovnkube-script-lib\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.437252 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435193 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-ovn\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.437252 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435193 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-systemd\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.437252 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435280 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.437252 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435441 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-env-overrides\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.437252 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435661 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2590e213-dc39-406a-8094-b4e01162e221-cni-binary-copy\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.437252 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435740 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/308feccf-7178-48c7-b370-3c78f96e8919-etc-tuned\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.437252 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435754 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b33b71b3-287e-4516-9d0a-726f302a8dff-iptables-alerter-script\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:56:23.437252 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435840 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovnkube-script-lib\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.437252 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435869 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovnkube-config\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.437252 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.435882 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/308feccf-7178-48c7-b370-3c78f96e8919-tmp\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.437252 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.436574 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovn-node-metrics-cert\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.437252 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.437136 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/642aaac2-4dc2-4800-bc55-c70730884400-agent-certs\") pod \"konnectivity-agent-9ktt5\" (UID: \"642aaac2-4dc2-4800-bc55-c70730884400\") " pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:56:23.445947 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.445920 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qspj\" (UniqueName: \"kubernetes.io/projected/b33b71b3-287e-4516-9d0a-726f302a8dff-kube-api-access-8qspj\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:56:23.446183 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.446162 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn92t\" (UniqueName: \"kubernetes.io/projected/33806d61-5ac2-49ba-9fc9-c89046f2ff03-kube-api-access-tn92t\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:23.446268 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.446201 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kwj4\" (UniqueName: \"kubernetes.io/projected/2590e213-dc39-406a-8094-b4e01162e221-kube-api-access-8kwj4\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.446579 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.446557 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lplg9\" (UniqueName: \"kubernetes.io/projected/308feccf-7178-48c7-b370-3c78f96e8919-kube-api-access-lplg9\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.447117 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.447102 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp952\" (UniqueName: \"kubernetes.io/projected/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-kube-api-access-fp952\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.447168 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.447138 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz2q2\" (UniqueName: \"kubernetes.io/projected/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-kube-api-access-wz2q2\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.447168 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.447148 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbht\" (UniqueName: \"kubernetes.io/projected/7f215e51-f5cf-4103-9904-693d5721e647-kube-api-access-mgbht\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:56:23.447487 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.447472 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hggp5\" (UniqueName: \"kubernetes.io/projected/ffa0c959-6bd1-4feb-b713-3545a6896409-kube-api-access-hggp5\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:56:23.504562 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.504538 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:23.564801 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.564738 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:56:23.567874 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.567851 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5p4wp" Nov 03 19:56:23.573749 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:23.573721 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c45b0c_9bd5_42bc_8a11_b2db15876d63.slice/crio-92b192f6ef7e7367637dfb3e494ac28951e607715d76272d1eb7024ef0e27aea WatchSource:0}: Error finding container 92b192f6ef7e7367637dfb3e494ac28951e607715d76272d1eb7024ef0e27aea: Status 404 returned error can't find the container with id 92b192f6ef7e7367637dfb3e494ac28951e607715d76272d1eb7024ef0e27aea Nov 03 19:56:23.576016 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:23.575996 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2590e213_dc39_406a_8094_b4e01162e221.slice/crio-51fd09239009d0c5bdd7f1f9ec686fb72c32f8abaf5538ca38165bf79a7ef043 WatchSource:0}: Error finding container 51fd09239009d0c5bdd7f1f9ec686fb72c32f8abaf5538ca38165bf79a7ef043: Status 404 returned error can't find the container with id 51fd09239009d0c5bdd7f1f9ec686fb72c32f8abaf5538ca38165bf79a7ef043 Nov 03 19:56:23.592967 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.592940 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:56:23.595207 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.595192 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:23.597364 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.597349 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:56:23.601013 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:23.600996 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33b71b3_287e_4516_9d0a_726f302a8dff.slice/crio-830e932e628998416cad3c6aa6444beb61b0ff8a2bad9f109e99ce1b1b23b19d WatchSource:0}: Error finding container 830e932e628998416cad3c6aa6444beb61b0ff8a2bad9f109e99ce1b1b23b19d: Status 404 returned error can't find the container with id 830e932e628998416cad3c6aa6444beb61b0ff8a2bad9f109e99ce1b1b23b19d Nov 03 19:56:23.605044 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:23.605027 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642aaac2_4dc2_4800_bc55_c70730884400.slice/crio-4b0347a2efe0acd92df23648fa9c9d6f64d992dac5e32646c46e123b2239f207 WatchSource:0}: Error finding container 4b0347a2efe0acd92df23648fa9c9d6f64d992dac5e32646c46e123b2239f207: Status 404 returned error can't find the container with id 4b0347a2efe0acd92df23648fa9c9d6f64d992dac5e32646c46e123b2239f207 Nov 03 19:56:23.611155 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.611140 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:56:23.618605 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:23.618588 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308feccf_7178_48c7_b370_3c78f96e8919.slice/crio-1f55a3cc823fa180c91e85c0ff91e271bf36758a451fe42a0be562c089a59108 WatchSource:0}: Error finding container 1f55a3cc823fa180c91e85c0ff91e271bf36758a451fe42a0be562c089a59108: Status 404 returned error can't find the container with id 1f55a3cc823fa180c91e85c0ff91e271bf36758a451fe42a0be562c089a59108 Nov 03 19:56:23.627990 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.627977 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:56:23.636411 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:23.636374 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa0c959_6bd1_4feb_b713_3545a6896409.slice/crio-7bc7577922eb0414c657d916675bb16b35a5ad6a22526abf21ba7aae8fc9775e WatchSource:0}: Error finding container 7bc7577922eb0414c657d916675bb16b35a5ad6a22526abf21ba7aae8fc9775e: Status 404 returned error can't find the container with id 7bc7577922eb0414c657d916675bb16b35a5ad6a22526abf21ba7aae8fc9775e Nov 03 19:56:23.643359 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.643341 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:56:23.652197 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:23.652179 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc075c5d_3e03_4e5c_be0d_09efa41fcdb9.slice/crio-6e334e0f5c5cb92cfdf99467adb29e5e8c2c6f51a68919dc8164e62ff7063339 WatchSource:0}: Error finding container 6e334e0f5c5cb92cfdf99467adb29e5e8c2c6f51a68919dc8164e62ff7063339: Status 404 returned error can't find the container with id 6e334e0f5c5cb92cfdf99467adb29e5e8c2c6f51a68919dc8164e62ff7063339 Nov 03 19:56:23.656953 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.656940 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:56:23.665120 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:23.665100 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f215e51_f5cf_4103_9904_693d5721e647.slice/crio-0a77dd79af69bc379beceb5011ac869c45b9e91bd7cfda7b3bfcca7b49b2e278 WatchSource:0}: Error finding container 0a77dd79af69bc379beceb5011ac869c45b9e91bd7cfda7b3bfcca7b49b2e278: Status 404 returned error can't find the container with id 0a77dd79af69bc379beceb5011ac869c45b9e91bd7cfda7b3bfcca7b49b2e278 Nov 03 19:56:23.674888 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.674872 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:23.683422 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:23.683401 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf954863_9eb8_4bd6_a64b_918f5f4ff6b0.slice/crio-c7e7902ba9bf3721dba8987ea143d83d5910a55d479f46a8fe049347e5508fb4 WatchSource:0}: Error finding container c7e7902ba9bf3721dba8987ea143d83d5910a55d479f46a8fe049347e5508fb4: Status 404 returned error can't find the container with id c7e7902ba9bf3721dba8987ea143d83d5910a55d479f46a8fe049347e5508fb4 Nov 03 19:56:23.937713 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.937636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:23.937713 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.937682 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qtn\" (UniqueName: \"kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn\") pod \"network-check-target-h4zrs\" (UID: \"c98e82ab-c3b3-4d18-b381-083fdb90b448\") " pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:23.937878 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:23.937720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:23.937878 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.937740 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:23.937878 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.937803 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret podName:e6338d09-ef6a-4cf5-8a06-bd466bc5076d nodeName:}" failed. No retries permitted until 2026-11-03 19:56:24.937785273 +0000 UTC m=+3.065514343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret") pod "global-pull-secret-syncer-l5wn9" (UID: "e6338d09-ef6a-4cf5-8a06-bd466bc5076d") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:23.937878 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.937803 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:23.937878 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.937840 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs podName:33806d61-5ac2-49ba-9fc9-c89046f2ff03 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:24.937830913 +0000 UTC m=+3.065559970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs") pod "network-metrics-daemon-st6tw" (UID: "33806d61-5ac2-49ba-9fc9-c89046f2ff03") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:23.937878 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.937850 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:23.937878 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.937863 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:23.937878 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.937873 2579 projected.go:196] Error preparing data for projected volume kube-api-access-75qtn for pod openshift-network-diagnostics/network-check-target-h4zrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:23.938165 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:23.937906 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn podName:c98e82ab-c3b3-4d18-b381-083fdb90b448 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:24.937896364 +0000 UTC m=+3.065625419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-75qtn" (UniqueName: "kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn") pod "network-check-target-h4zrs" (UID: "c98e82ab-c3b3-4d18-b381-083fdb90b448") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:24.229203 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.229130 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:24.267585 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.267476 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:51:23 +0000 UTC" deadline="2028-05-03 04:09:28.570269278 +0000 UTC" Nov 03 19:56:24.267745 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.267594 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13112h13m4.302689731s" Nov 03 19:56:24.364133 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.364081 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" event={"ID":"af954863-9eb8-4bd6-a64b-918f5f4ff6b0","Type":"ContainerStarted","Data":"c7e7902ba9bf3721dba8987ea143d83d5910a55d479f46a8fe049347e5508fb4"} Nov 03 19:56:24.385811 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.385777 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ssj5l" event={"ID":"7f215e51-f5cf-4103-9904-693d5721e647","Type":"ContainerStarted","Data":"0a77dd79af69bc379beceb5011ac869c45b9e91bd7cfda7b3bfcca7b49b2e278"} Nov 03 19:56:24.401844 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.401780 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" event={"ID":"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9","Type":"ContainerStarted","Data":"6e334e0f5c5cb92cfdf99467adb29e5e8c2c6f51a68919dc8164e62ff7063339"} Nov 03 19:56:24.409064 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.409035 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cwtc9" event={"ID":"ffa0c959-6bd1-4feb-b713-3545a6896409","Type":"ContainerStarted","Data":"7bc7577922eb0414c657d916675bb16b35a5ad6a22526abf21ba7aae8fc9775e"} Nov 03 19:56:24.412161 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.412089 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q5hct" event={"ID":"308feccf-7178-48c7-b370-3c78f96e8919","Type":"ContainerStarted","Data":"1f55a3cc823fa180c91e85c0ff91e271bf36758a451fe42a0be562c089a59108"} Nov 03 19:56:24.420800 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.420776 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zqftx" event={"ID":"b33b71b3-287e-4516-9d0a-726f302a8dff","Type":"ContainerStarted","Data":"830e932e628998416cad3c6aa6444beb61b0ff8a2bad9f109e99ce1b1b23b19d"} Nov 03 19:56:24.423027 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.422959 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5p4wp" event={"ID":"2590e213-dc39-406a-8094-b4e01162e221","Type":"ContainerStarted","Data":"51fd09239009d0c5bdd7f1f9ec686fb72c32f8abaf5538ca38165bf79a7ef043"} Nov 03 19:56:24.427729 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.427709 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" event={"ID":"ca7a77d780aa380ba43b79db4d32277a","Type":"ContainerStarted","Data":"35150edf568d96bc082d83b80c1db26cdb6a69c1259c86011542b4cc61f141a6"} Nov 03 19:56:24.436785 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.435945 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9ktt5" event={"ID":"642aaac2-4dc2-4800-bc55-c70730884400","Type":"ContainerStarted","Data":"4b0347a2efe0acd92df23648fa9c9d6f64d992dac5e32646c46e123b2239f207"} Nov 03 19:56:24.438745 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.438698 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx4kz" event={"ID":"c2c45b0c-9bd5-42bc-8a11-b2db15876d63","Type":"ContainerStarted","Data":"92b192f6ef7e7367637dfb3e494ac28951e607715d76272d1eb7024ef0e27aea"} Nov 03 19:56:24.441064 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.441007 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" event={"ID":"0616d8478d645504ab20b6a243f25dbb","Type":"ContainerStarted","Data":"bd6773b2f462d346511693cad2335a5ba671730d1886dde1e05ec240083a3169"} Nov 03 19:56:24.947160 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.946753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:24.947160 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.946821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:24.947160 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:24.946845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qtn\" (UniqueName: \"kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn\") pod \"network-check-target-h4zrs\" (UID: \"c98e82ab-c3b3-4d18-b381-083fdb90b448\") " pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:24.947160 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:24.946931 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:24.947160 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:24.946968 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:24.947160 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:24.946983 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:24.947160 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:24.946996 2579 projected.go:196] Error preparing data for projected volume kube-api-access-75qtn for pod openshift-network-diagnostics/network-check-target-h4zrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:24.947160 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:24.947002 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs podName:33806d61-5ac2-49ba-9fc9-c89046f2ff03 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:26.946983984 +0000 UTC m=+5.074713059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs") pod "network-metrics-daemon-st6tw" (UID: "33806d61-5ac2-49ba-9fc9-c89046f2ff03") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:24.947160 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:24.947041 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn podName:c98e82ab-c3b3-4d18-b381-083fdb90b448 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:26.947026671 +0000 UTC m=+5.074755730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-75qtn" (UniqueName: "kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn") pod "network-check-target-h4zrs" (UID: "c98e82ab-c3b3-4d18-b381-083fdb90b448") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:24.947160 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:24.947104 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:24.947160 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:24.947137 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret podName:e6338d09-ef6a-4cf5-8a06-bd466bc5076d nodeName:}" failed. No retries permitted until 2026-11-03 19:56:26.947125693 +0000 UTC m=+5.074854751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret") pod "global-pull-secret-syncer-l5wn9" (UID: "e6338d09-ef6a-4cf5-8a06-bd466bc5076d") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:25.268575 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:25.268447 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:51:23 +0000 UTC" deadline="2028-04-20 16:14:52.012239621 +0000 UTC" Nov 03 19:56:25.268575 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:25.268489 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12812h18m26.743755096s" Nov 03 19:56:25.355144 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:25.354455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:25.355144 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:25.354652 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:25.355144 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:25.354916 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:25.355144 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:25.355003 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:25.355144 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:25.355032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:25.355144 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:25.355112 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:25.697373 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:25.697298 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:56:26.967140 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:26.967062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:26.967140 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:26.967114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qtn\" (UniqueName: \"kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn\") pod \"network-check-target-h4zrs\" (UID: \"c98e82ab-c3b3-4d18-b381-083fdb90b448\") " pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:26.967770 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:26.967156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:26.967770 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:26.967316 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:26.967770 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:26.967377 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs podName:33806d61-5ac2-49ba-9fc9-c89046f2ff03 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:30.967360446 +0000 UTC m=+9.095089519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs") pod "network-metrics-daemon-st6tw" (UID: "33806d61-5ac2-49ba-9fc9-c89046f2ff03") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:26.967930 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:26.967781 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:26.967930 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:26.967827 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret podName:e6338d09-ef6a-4cf5-8a06-bd466bc5076d nodeName:}" failed. No retries permitted until 2026-11-03 19:56:30.967813385 +0000 UTC m=+9.095542456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret") pod "global-pull-secret-syncer-l5wn9" (UID: "e6338d09-ef6a-4cf5-8a06-bd466bc5076d") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:26.967930 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:26.967901 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:26.967930 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:26.967915 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:26.967930 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:26.967927 2579 projected.go:196] Error preparing data for projected volume kube-api-access-75qtn for pod openshift-network-diagnostics/network-check-target-h4zrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:26.968161 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:26.967974 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn podName:c98e82ab-c3b3-4d18-b381-083fdb90b448 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:30.967962637 +0000 UTC m=+9.095691701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-75qtn" (UniqueName: "kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn") pod "network-check-target-h4zrs" (UID: "c98e82ab-c3b3-4d18-b381-083fdb90b448") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:27.354621 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:27.354430 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:27.354782 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:27.354730 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:27.355131 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:27.355113 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:27.355255 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:27.355223 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:27.355312 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:27.355286 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:27.355369 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:27.355352 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:27.474427 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:27.473564 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx4kz" event={"ID":"c2c45b0c-9bd5-42bc-8a11-b2db15876d63","Type":"ContainerStarted","Data":"b83017975f7dfe5b6b6da6d7bde2ba5d209b522d5818aae9e397682e9294eae0"} Nov 03 19:56:27.479845 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:27.479228 2579 generic.go:356] "Generic (PLEG): container finished" podID="0616d8478d645504ab20b6a243f25dbb" containerID="589741ed9276485d832c4b782d1b14eb59a4c5ac86f2741859bda65c8afc1faf" exitCode=0 Nov 03 19:56:27.479845 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:27.479271 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" event={"ID":"0616d8478d645504ab20b6a243f25dbb","Type":"ContainerDied","Data":"589741ed9276485d832c4b782d1b14eb59a4c5ac86f2741859bda65c8afc1faf"} Nov 03 19:56:28.485318 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:28.485249 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" event={"ID":"ca7a77d780aa380ba43b79db4d32277a","Type":"ContainerStarted","Data":"7cd8ef78a2b0513babcc7e551998130cd6b72424cdcd67acaa1d02e70cef2fdc"} Nov 03 19:56:28.499572 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:28.499538 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9ktt5" event={"ID":"642aaac2-4dc2-4800-bc55-c70730884400","Type":"ContainerStarted","Data":"c301e1e47efac2e9e61069e0ba712fb5d4a40e0b4b9d9b2bf138046c6475b367"} Nov 03 19:56:28.504410 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:28.504358 2579 generic.go:356] "Generic (PLEG): container finished" podID="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" containerID="b83017975f7dfe5b6b6da6d7bde2ba5d209b522d5818aae9e397682e9294eae0" exitCode=0 Nov 03 19:56:28.504545 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:28.504449 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx4kz" event={"ID":"c2c45b0c-9bd5-42bc-8a11-b2db15876d63","Type":"ContainerDied","Data":"b83017975f7dfe5b6b6da6d7bde2ba5d209b522d5818aae9e397682e9294eae0"} Nov 03 19:56:28.518690 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:28.518644 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" event={"ID":"0616d8478d645504ab20b6a243f25dbb","Type":"ContainerStarted","Data":"d74fc62c5607f6b02ce6ae75511f0601c705ab8aae3ebf8548121158022b8c10"} Nov 03 19:56:28.528678 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:28.528629 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" podStartSLOduration=6.528613923 podStartE2EDuration="6.528613923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:28.504329479 +0000 UTC m=+6.632058561" watchObservedRunningTime="2026-11-03 19:56:28.528613923 +0000 UTC m=+6.656343003" Nov 03 19:56:28.548188 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:28.548118 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9ktt5" podStartSLOduration=2.30975249 podStartE2EDuration="6.548099297s" totalImagesPullingTime="4.238346807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:28.529249373 +0000 UTC m=+6.656978441" watchObservedRunningTime="2026-11-03 19:56:28.548099297 +0000 UTC m=+6.675828379" Nov 03 19:56:28.548915 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:28.548860 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" podStartSLOduration=6.548847274 podStartE2EDuration="6.548847274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:28.54789965 +0000 UTC m=+6.675628730" watchObservedRunningTime="2026-11-03 19:56:28.548847274 +0000 UTC m=+6.676576354" Nov 03 19:56:29.355030 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:29.354659 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:29.355030 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:29.354803 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:29.355225 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:29.355178 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:29.355892 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:29.355264 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:29.355892 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:29.355316 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:29.355892 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:29.355381 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:29.532718 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:29.532688 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cwtc9" event={"ID":"ffa0c959-6bd1-4feb-b713-3545a6896409","Type":"ContainerStarted","Data":"41d64f41dcbaaf24710db237dfef41d4d4a2a4553f0aabf507fc33bc432e8b42"} Nov 03 19:56:30.536421 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:30.536319 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" event={"ID":"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9","Type":"ContainerStarted","Data":"5fe0013d85daa4974293650a61121c8ddd21a12bfd9734d65992ca7f7d1e0c2e"} Nov 03 19:56:31.008639 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:31.007651 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:31.008639 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:31.007748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:31.008639 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:31.007778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qtn\" (UniqueName: \"kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn\") pod \"network-check-target-h4zrs\" (UID: \"c98e82ab-c3b3-4d18-b381-083fdb90b448\") " pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:31.008639 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:31.007930 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:31.008639 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:31.007950 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:31.008639 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:31.007964 2579 projected.go:196] Error preparing data for projected volume kube-api-access-75qtn for pod openshift-network-diagnostics/network-check-target-h4zrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:31.008639 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:31.008020 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn podName:c98e82ab-c3b3-4d18-b381-083fdb90b448 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:39.008004098 +0000 UTC m=+17.135733175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-75qtn" (UniqueName: "kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn") pod "network-check-target-h4zrs" (UID: "c98e82ab-c3b3-4d18-b381-083fdb90b448") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:31.008639 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:31.008446 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:31.008639 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:31.008501 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs podName:33806d61-5ac2-49ba-9fc9-c89046f2ff03 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:39.008485903 +0000 UTC m=+17.136214976 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs") pod "network-metrics-daemon-st6tw" (UID: "33806d61-5ac2-49ba-9fc9-c89046f2ff03") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:31.008639 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:31.008566 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:31.008639 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:31.008600 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret podName:e6338d09-ef6a-4cf5-8a06-bd466bc5076d nodeName:}" failed. No retries permitted until 2026-11-03 19:56:39.008589172 +0000 UTC m=+17.136318261 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret") pod "global-pull-secret-syncer-l5wn9" (UID: "e6338d09-ef6a-4cf5-8a06-bd466bc5076d") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:31.355615 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:31.354806 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:31.355615 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:31.354936 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:31.355615 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:31.355009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:31.355615 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:31.355098 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:31.355615 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:31.355149 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:31.355615 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:31.355220 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:31.563611 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:31.563576 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:56:32.295584 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:32.295484 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:56:31.563595608Z","UUID":"7f1ba79e-2205-4b82-97b5-676c63c63bac","Handler":null,"Name":"","Endpoint":""} Nov 03 19:56:32.301314 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:32.301247 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:56:32.301314 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:32.301332 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:56:32.546730 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:32.546524 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ssj5l" event={"ID":"7f215e51-f5cf-4103-9904-693d5721e647","Type":"ContainerStarted","Data":"e2a49d41da755d90496ffcb584a8d137195ef89a83a23e77a1d709106fe70fca"} Nov 03 19:56:32.548765 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:32.548736 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" event={"ID":"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9","Type":"ContainerStarted","Data":"c7782cd5b1fd6b25b60db1edf94d55148885890aa2c6bcda474d6e242f4adc76"} Nov 03 19:56:32.550671 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:32.550642 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5p4wp" event={"ID":"2590e213-dc39-406a-8094-b4e01162e221","Type":"ContainerStarted","Data":"b53c7a919e55bd55441dcb01c3892910ed6510b7a9f41c0663d1c488772e80de"} Nov 03 19:56:32.552548 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:32.552518 2579 generic.go:356] "Generic (PLEG): container finished" podID="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" containerID="3ebfd424d879d07593d5bdcfb82f532fbee4a760f4180e3af4e6cc6517657154" exitCode=0 Nov 03 19:56:32.552624 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:32.552556 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx4kz" event={"ID":"c2c45b0c-9bd5-42bc-8a11-b2db15876d63","Type":"ContainerDied","Data":"3ebfd424d879d07593d5bdcfb82f532fbee4a760f4180e3af4e6cc6517657154"} Nov 03 19:56:32.564014 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:32.563966 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ssj5l" podStartSLOduration=2.21057787 podStartE2EDuration="10.563949696s" totalImagesPullingTime="8.353371826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:32.563930057 +0000 UTC m=+10.691659138" watchObservedRunningTime="2026-11-03 19:56:32.563949696 +0000 UTC m=+10.691678809" Nov 03 19:56:32.564653 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:32.564616 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cwtc9" podStartSLOduration=5.165549358 podStartE2EDuration="10.564606038s" totalImagesPullingTime="5.39905668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:29.549190141 +0000 UTC m=+7.676919221" watchObservedRunningTime="2026-11-03 19:56:32.564606038 +0000 UTC m=+10.692335117" Nov 03 19:56:32.583900 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:32.583840 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5p4wp" podStartSLOduration=2.657678063 podStartE2EDuration="10.583824335s" totalImagesPullingTime="7.926146272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:32.583346268 +0000 UTC m=+10.711075347" watchObservedRunningTime="2026-11-03 19:56:32.583824335 +0000 UTC m=+10.711553415" Nov 03 19:56:33.343986 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:33.343950 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:56:33.345263 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:33.345240 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:56:33.354501 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:33.354474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:33.354606 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:33.354504 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:33.354606 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:33.354481 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:33.354721 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:33.354602 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:33.354721 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:33.354684 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:33.354816 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:33.354742 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:33.558078 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:33.558001 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zqftx" event={"ID":"b33b71b3-287e-4516-9d0a-726f302a8dff","Type":"ContainerStarted","Data":"30313a8b2b236615c7112887555668f6201b1025e327eb842f0e6603a872d4e0"} Nov 03 19:56:33.561791 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:33.561752 2579 generic.go:356] "Generic (PLEG): container finished" podID="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" containerID="eedef9467086f2cdc20e645a9a90b0dc2d310c32789643a0586767fbf9d73f87" exitCode=0 Nov 03 19:56:33.561967 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:33.561840 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx4kz" event={"ID":"c2c45b0c-9bd5-42bc-8a11-b2db15876d63","Type":"ContainerDied","Data":"eedef9467086f2cdc20e645a9a90b0dc2d310c32789643a0586767fbf9d73f87"} Nov 03 19:56:33.565790 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:33.565764 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" event={"ID":"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9","Type":"ContainerStarted","Data":"c49a6e3412335879702a61127b2adc0eca8c5cbf82c9cb9a38144154bec26364"} Nov 03 19:56:33.574607 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:33.574564 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zqftx" podStartSLOduration=3.378982041 podStartE2EDuration="11.574551327s" totalImagesPullingTime="8.195569286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:33.574486237 +0000 UTC m=+11.702215317" watchObservedRunningTime="2026-11-03 19:56:33.574551327 +0000 UTC m=+11.702280406" Nov 03 19:56:33.617796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:33.617731 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" podStartSLOduration=3.020523146 podStartE2EDuration="11.617712082s" totalImagesPullingTime="8.597188936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:33.617247061 +0000 UTC m=+11.744976143" watchObservedRunningTime="2026-11-03 19:56:33.617712082 +0000 UTC m=+11.745441165" Nov 03 19:56:34.569096 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:34.569061 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q5hct" event={"ID":"308feccf-7178-48c7-b370-3c78f96e8919","Type":"ContainerStarted","Data":"d85937c09483a292e3db983a40a78d361c57d11cfe671657896ccafbc6cc9645"} Nov 03 19:56:34.571957 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:34.571929 2579 generic.go:356] "Generic (PLEG): container finished" podID="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" containerID="9273dda4508acde0057b732234ef22156a38fbab9bd8c579c9d712cf72d55cf6" exitCode=0 Nov 03 19:56:34.572071 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:34.572011 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx4kz" event={"ID":"c2c45b0c-9bd5-42bc-8a11-b2db15876d63","Type":"ContainerDied","Data":"9273dda4508acde0057b732234ef22156a38fbab9bd8c579c9d712cf72d55cf6"} Nov 03 19:56:34.588187 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:34.588146 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q5hct" podStartSLOduration=2.393290779 podStartE2EDuration="12.588132922s" totalImagesPullingTime="10.194842143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:34.587495427 +0000 UTC m=+12.715224506" watchObservedRunningTime="2026-11-03 19:56:34.588132922 +0000 UTC m=+12.715862000" Nov 03 19:56:35.354535 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:35.354497 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:35.354682 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:35.354640 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:35.354743 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:35.354716 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:35.354775 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:35.354762 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:35.354823 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:35.354805 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:35.354947 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:35.354861 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:36.580547 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:36.580318 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx4kz" event={"ID":"c2c45b0c-9bd5-42bc-8a11-b2db15876d63","Type":"ContainerStarted","Data":"8400ed3b6e43b5ae4f65167de453975dc398ce9dbf85accb0c5392c9a6b34db8"} Nov 03 19:56:36.735000 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:36.734904 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:56:36.735151 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:36.735020 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:56:36.735819 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:36.735795 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:56:37.354137 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:37.354043 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:37.354137 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:37.354081 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:37.354137 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:37.354081 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:37.354430 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:37.354174 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:37.354430 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:37.354246 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:37.354430 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:37.354351 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:37.586361 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:37.586326 2579 generic.go:356] "Generic (PLEG): container finished" podID="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" containerID="8400ed3b6e43b5ae4f65167de453975dc398ce9dbf85accb0c5392c9a6b34db8" exitCode=0 Nov 03 19:56:37.586842 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:37.586478 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx4kz" event={"ID":"c2c45b0c-9bd5-42bc-8a11-b2db15876d63","Type":"ContainerDied","Data":"8400ed3b6e43b5ae4f65167de453975dc398ce9dbf85accb0c5392c9a6b34db8"} Nov 03 19:56:38.591244 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:38.591207 2579 generic.go:356] "Generic (PLEG): container finished" podID="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" containerID="8511e63cd28397e00df69b9a31c960cc21849edc7409cbe9d4f258ed2b709c9b" exitCode=0 Nov 03 19:56:38.591620 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:38.591249 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx4kz" event={"ID":"c2c45b0c-9bd5-42bc-8a11-b2db15876d63","Type":"ContainerDied","Data":"8511e63cd28397e00df69b9a31c960cc21849edc7409cbe9d4f258ed2b709c9b"} Nov 03 19:56:39.068114 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.068089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:39.068233 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.068121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qtn\" (UniqueName: \"kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn\") pod \"network-check-target-h4zrs\" (UID: \"c98e82ab-c3b3-4d18-b381-083fdb90b448\") " pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:39.068233 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.068145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:39.068340 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:39.068230 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:39.068340 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:39.068240 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:39.068340 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:39.068245 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 19:56:39.068340 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:39.068260 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 19:56:39.068340 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:39.068272 2579 projected.go:196] Error preparing data for projected volume kube-api-access-75qtn for pod openshift-network-diagnostics/network-check-target-h4zrs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:39.068340 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:39.068293 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs podName:33806d61-5ac2-49ba-9fc9-c89046f2ff03 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:55.068276277 +0000 UTC m=+33.196005351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs") pod "network-metrics-daemon-st6tw" (UID: "33806d61-5ac2-49ba-9fc9-c89046f2ff03") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 19:56:39.068340 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:39.068309 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn podName:c98e82ab-c3b3-4d18-b381-083fdb90b448 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:55.06830113 +0000 UTC m=+33.196030195 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-75qtn" (UniqueName: "kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn") pod "network-check-target-h4zrs" (UID: "c98e82ab-c3b3-4d18-b381-083fdb90b448") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 19:56:39.068340 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:39.068324 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret podName:e6338d09-ef6a-4cf5-8a06-bd466bc5076d nodeName:}" failed. No retries permitted until 2026-11-03 19:56:55.068316027 +0000 UTC m=+33.196045091 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret") pod "global-pull-secret-syncer-l5wn9" (UID: "e6338d09-ef6a-4cf5-8a06-bd466bc5076d") : object "kube-system"/"original-pull-secret" not registered Nov 03 19:56:39.354462 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.354284 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:39.354575 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.354339 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:39.354693 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:39.354669 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:39.354737 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:39.354545 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:39.354737 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.354339 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:39.354801 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:39.354783 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:39.596184 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.596113 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 19:56:39.596554 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.596441 2579 generic.go:356] "Generic (PLEG): container finished" podID="af954863-9eb8-4bd6-a64b-918f5f4ff6b0" containerID="a000026611a4514c07cd1cc9067e28b492e536d864da11d11971173f90384cbb" exitCode=1 Nov 03 19:56:39.596554 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.596508 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" event={"ID":"af954863-9eb8-4bd6-a64b-918f5f4ff6b0","Type":"ContainerStarted","Data":"17cd414dee9c6e95d6e2aa4894fdc9d25aebea83bd43052993ce67f8a25584e3"} Nov 03 19:56:39.596637 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.596599 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" event={"ID":"af954863-9eb8-4bd6-a64b-918f5f4ff6b0","Type":"ContainerStarted","Data":"7cb315856c844467f7891832e3683a6cca4cfad51c8d004bc6bd41127e60dc68"} Nov 03 19:56:39.596637 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.596629 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" event={"ID":"af954863-9eb8-4bd6-a64b-918f5f4ff6b0","Type":"ContainerStarted","Data":"95f312370434529e87050a0112521824d8259f53e75f1ec3323df12166f11e06"} Nov 03 19:56:39.596703 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.596655 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" event={"ID":"af954863-9eb8-4bd6-a64b-918f5f4ff6b0","Type":"ContainerStarted","Data":"ac5e5726f6d582f9c1fcb91b0631a6b506125ee1597b6f3ef7819e10e441467a"} Nov 03 19:56:39.596703 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.596683 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" event={"ID":"af954863-9eb8-4bd6-a64b-918f5f4ff6b0","Type":"ContainerDied","Data":"a000026611a4514c07cd1cc9067e28b492e536d864da11d11971173f90384cbb"} Nov 03 19:56:39.596771 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.596719 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" event={"ID":"af954863-9eb8-4bd6-a64b-918f5f4ff6b0","Type":"ContainerStarted","Data":"88ff826a54ae85596e2a0a7cb246a9955c8f0b47191308a4d4f202d5c15e49bb"} Nov 03 19:56:39.599492 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.599466 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xx4kz" event={"ID":"c2c45b0c-9bd5-42bc-8a11-b2db15876d63","Type":"ContainerStarted","Data":"db78804adfbf2fbb574d8a51aad3be33685a91046e4851352c854ce35087db73"} Nov 03 19:56:39.626003 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:39.625966 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xx4kz" podStartSLOduration=6.686782938 podStartE2EDuration="17.625953886s" totalImagesPullingTime="10.939170948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:39.624745941 +0000 UTC m=+17.752475018" watchObservedRunningTime="2026-11-03 19:56:39.625953886 +0000 UTC m=+17.753682956" Nov 03 19:56:41.354539 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:41.354504 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:41.355009 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:41.354569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:41.355009 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:41.354656 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:41.355009 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:41.354667 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:41.355009 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:41.354735 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:41.355009 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:41.354817 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:41.606796 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:41.606724 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 19:56:41.607060 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:41.607039 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" event={"ID":"af954863-9eb8-4bd6-a64b-918f5f4ff6b0","Type":"ContainerStarted","Data":"1ab04fc414b5a43a8379835f114e624e64b39c95a16ffce37b94c04d995bebcc"} Nov 03 19:56:43.353885 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:43.353838 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:43.353885 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:43.353872 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:43.354378 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:43.353859 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:43.354378 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:43.353952 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:43.354378 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:43.354018 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:43.354378 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:43.354075 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:43.616269 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:43.616203 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 19:56:43.616664 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:43.616625 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" event={"ID":"af954863-9eb8-4bd6-a64b-918f5f4ff6b0","Type":"ContainerStarted","Data":"e40fc369fc417f36a8dda1d04ccfaea6a6b08d1f58228633815ca54c9bc9d502"} Nov 03 19:56:43.617852 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:43.617819 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:43.617992 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:43.617979 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:43.618169 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:43.618116 2579 scope.go:122] "RemoveContainer" containerID="a000026611a4514c07cd1cc9067e28b492e536d864da11d11971173f90384cbb" Nov 03 19:56:43.636715 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:43.636692 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:43.637857 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:43.637834 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:44.622970 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:44.622943 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 19:56:44.623729 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:44.623250 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" event={"ID":"af954863-9eb8-4bd6-a64b-918f5f4ff6b0","Type":"ContainerStarted","Data":"d1c2986f302e168e14071386f6190495426e905bf0afde15aefba6a299e979e3"} Nov 03 19:56:44.623729 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:44.623341 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:56:44.660954 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:44.660888 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" podStartSLOduration=7.4277043240000005 podStartE2EDuration="22.660867451s" totalImagesPullingTime="15.233163127s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 19:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:44.659170823 +0000 UTC m=+22.786899903" watchObservedRunningTime="2026-11-03 19:56:44.660867451 +0000 UTC m=+22.788596531" Nov 03 19:56:45.225423 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:45.225145 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4zrs"] Nov 03 19:56:45.225552 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:45.225464 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:45.225607 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:45.225584 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:45.232618 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:45.232594 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l5wn9"] Nov 03 19:56:45.232722 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:45.232708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:45.232806 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:45.232790 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:45.233159 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:45.233107 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-st6tw"] Nov 03 19:56:45.233263 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:45.233212 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:45.233335 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:45.233314 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:45.625564 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:45.625496 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:56:47.354249 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:47.354214 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:47.354738 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:47.354231 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:47.354738 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:47.354337 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:47.354738 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:47.354380 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:47.354738 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:47.354448 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:47.354738 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:47.354337 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:49.354101 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:49.354070 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:49.354536 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:49.354078 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:49.354536 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:49.354168 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:49.354536 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:49.354198 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:49.354536 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:49.354296 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:49.354536 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:49.354421 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:50.397182 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:50.397152 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:50.397586 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:50.397344 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:56:50.408734 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:50.408712 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:56:51.354321 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:51.354288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:51.354505 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:51.354353 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:51.354505 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:51.354376 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:51.354596 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:51.354498 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-h4zrs" podUID="c98e82ab-c3b3-4d18-b381-083fdb90b448" Nov 03 19:56:51.354596 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:51.354545 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5wn9" podUID="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" Nov 03 19:56:51.354663 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:51.354615 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-st6tw" podUID="33806d61-5ac2-49ba-9fc9-c89046f2ff03" Nov 03 19:56:52.249382 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.249355 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeReady" Nov 03 19:56:52.249861 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.249515 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 19:56:52.303951 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.303903 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qxxzr"] Nov 03 19:56:52.350846 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.350811 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5zxlx"] Nov 03 19:56:52.351006 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.350899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxxzr" Nov 03 19:56:52.356011 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.355989 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 19:56:52.356133 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.356035 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 19:56:52.360675 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.360655 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 19:56:52.360783 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.360698 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rpk57\"" Nov 03 19:56:52.364260 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.364240 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxxzr"] Nov 03 19:56:52.364362 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.364269 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5zxlx"] Nov 03 19:56:52.364362 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.364295 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6tmzf"] Nov 03 19:56:52.364362 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.364335 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:52.367834 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.367816 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.371406 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.371368 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 19:56:52.371741 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.371721 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4mwrd\"" Nov 03 19:56:52.372231 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.372214 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kfdzx\"" Nov 03 19:56:52.372294 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.372249 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 19:56:52.374178 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.374161 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6tmzf"] Nov 03 19:56:52.456271 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.456239 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 19:56:52.456437 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.456334 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 19:56:52.456437 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.456349 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 19:56:52.456559 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.456546 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 19:56:52.471336 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.471315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqljf\" (UniqueName: \"kubernetes.io/projected/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-kube-api-access-dqljf\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:52.471462 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.471345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-config-volume\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:52.471462 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.471380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.471535 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.471504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-tmp-dir\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:52.471535 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.471523 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm4bm\" (UniqueName: \"kubernetes.io/projected/ac15ca75-0dd3-4f05-b21d-1ae2280d0415-kube-api-access-qm4bm\") pod \"ingress-canary-qxxzr\" (UID: \"ac15ca75-0dd3-4f05-b21d-1ae2280d0415\") " pod="openshift-ingress-canary/ingress-canary-qxxzr" Nov 03 19:56:52.471608 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.471544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.471608 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.471558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvxvg\" (UniqueName: \"kubernetes.io/projected/224062e4-7b08-4d73-ba03-1789a9f591be-kube-api-access-nvxvg\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.471608 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.471586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:52.471608 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.471602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/224062e4-7b08-4d73-ba03-1789a9f591be-data-volume\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.471741 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.471704 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac15ca75-0dd3-4f05-b21d-1ae2280d0415-cert\") pod \"ingress-canary-qxxzr\" (UID: \"ac15ca75-0dd3-4f05-b21d-1ae2280d0415\") " pod="openshift-ingress-canary/ingress-canary-qxxzr" Nov 03 19:56:52.471741 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.471726 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/224062e4-7b08-4d73-ba03-1789a9f591be-crio-socket\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.572154 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-tmp-dir\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:52.572154 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm4bm\" (UniqueName: \"kubernetes.io/projected/ac15ca75-0dd3-4f05-b21d-1ae2280d0415-kube-api-access-qm4bm\") pod \"ingress-canary-qxxzr\" (UID: \"ac15ca75-0dd3-4f05-b21d-1ae2280d0415\") " pod="openshift-ingress-canary/ingress-canary-qxxzr" Nov 03 19:56:52.572436 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.572436 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572194 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvxvg\" (UniqueName: \"kubernetes.io/projected/224062e4-7b08-4d73-ba03-1789a9f591be-kube-api-access-nvxvg\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.572436 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:52.572436 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572228 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/224062e4-7b08-4d73-ba03-1789a9f591be-data-volume\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.572436 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac15ca75-0dd3-4f05-b21d-1ae2280d0415-cert\") pod \"ingress-canary-qxxzr\" (UID: \"ac15ca75-0dd3-4f05-b21d-1ae2280d0415\") " pod="openshift-ingress-canary/ingress-canary-qxxzr" Nov 03 19:56:52.572436 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:52.572285 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:56:52.572436 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:52.572329 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:56:52.572436 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:52.572356 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm podName:224062e4-7b08-4d73-ba03-1789a9f591be nodeName:}" failed. No retries permitted until 2026-11-03 19:56:53.072337227 +0000 UTC m=+31.200066302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6tmzf" (UID: "224062e4-7b08-4d73-ba03-1789a9f591be") : configmap "kube-rbac-proxy" not found Nov 03 19:56:52.572436 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/224062e4-7b08-4d73-ba03-1789a9f591be-crio-socket\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.572436 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:52.572376 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls podName:731a10e2-8c1d-48c4-a453-1d1d589f6ae2 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:53.072366511 +0000 UTC m=+31.200095571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls") pod "dns-default-5zxlx" (UID: "731a10e2-8c1d-48c4-a453-1d1d589f6ae2") : secret "dns-default-metrics-tls" not found Nov 03 19:56:52.572436 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572436 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqljf\" (UniqueName: \"kubernetes.io/projected/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-kube-api-access-dqljf\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:52.572861 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572464 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/224062e4-7b08-4d73-ba03-1789a9f591be-crio-socket\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.572861 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-config-volume\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:52.572861 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.572861 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572533 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-tmp-dir\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:52.572861 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:52.572612 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:56:52.572861 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:52.572679 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls podName:224062e4-7b08-4d73-ba03-1789a9f591be nodeName:}" failed. No retries permitted until 2026-11-03 19:56:53.072661592 +0000 UTC m=+31.200390653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tmzf" (UID: "224062e4-7b08-4d73-ba03-1789a9f591be") : secret "insights-runtime-extractor-tls" not found Nov 03 19:56:52.572861 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572856 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/224062e4-7b08-4d73-ba03-1789a9f591be-data-volume\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.573058 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.572931 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-config-volume\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:52.576018 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.575998 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac15ca75-0dd3-4f05-b21d-1ae2280d0415-cert\") pod \"ingress-canary-qxxzr\" (UID: \"ac15ca75-0dd3-4f05-b21d-1ae2280d0415\") " pod="openshift-ingress-canary/ingress-canary-qxxzr" Nov 03 19:56:52.584417 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.584396 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqljf\" (UniqueName: \"kubernetes.io/projected/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-kube-api-access-dqljf\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:52.584518 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.584460 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvxvg\" (UniqueName: \"kubernetes.io/projected/224062e4-7b08-4d73-ba03-1789a9f591be-kube-api-access-nvxvg\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:52.584620 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.584604 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm4bm\" (UniqueName: \"kubernetes.io/projected/ac15ca75-0dd3-4f05-b21d-1ae2280d0415-kube-api-access-qm4bm\") pod \"ingress-canary-qxxzr\" (UID: \"ac15ca75-0dd3-4f05-b21d-1ae2280d0415\") " pod="openshift-ingress-canary/ingress-canary-qxxzr" Nov 03 19:56:52.666577 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.666547 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qxxzr" Nov 03 19:56:52.846374 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.846314 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zfvf2"] Nov 03 19:56:52.864541 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.864520 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zfvf2"] Nov 03 19:56:52.864653 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.864608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:56:52.868462 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.868410 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 19:56:52.868462 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.868423 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 19:56:52.868594 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.868480 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 19:56:52.868594 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.868490 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 19:56:52.868594 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.868552 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-f759x\"" Nov 03 19:56:52.976442 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.976403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97d28296-595f-4a77-926d-f6c30841085f-signing-key\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:56:52.976442 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.976444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97d28296-595f-4a77-926d-f6c30841085f-signing-cabundle\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:56:52.976634 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:52.976488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czx85\" (UniqueName: \"kubernetes.io/projected/97d28296-595f-4a77-926d-f6c30841085f-kube-api-access-czx85\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:56:53.010666 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.010634 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qxxzr"] Nov 03 19:56:53.013431 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:53.013406 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac15ca75_0dd3_4f05_b21d_1ae2280d0415.slice/crio-82bc135f2f1029b30e6ffb0ff31aeee61cd7290ca6797a6b023ab1b942cf88a0 WatchSource:0}: Error finding container 82bc135f2f1029b30e6ffb0ff31aeee61cd7290ca6797a6b023ab1b942cf88a0: Status 404 returned error can't find the container with id 82bc135f2f1029b30e6ffb0ff31aeee61cd7290ca6797a6b023ab1b942cf88a0 Nov 03 19:56:53.077894 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.077862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:53.078067 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.077920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:53.078067 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:53.078026 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:56:53.078067 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.078048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97d28296-595f-4a77-926d-f6c30841085f-signing-key\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:56:53.078187 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:53.078026 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:56:53.078187 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:53.078089 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls podName:224062e4-7b08-4d73-ba03-1789a9f591be nodeName:}" failed. No retries permitted until 2026-11-03 19:56:54.078071054 +0000 UTC m=+32.205800116 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tmzf" (UID: "224062e4-7b08-4d73-ba03-1789a9f591be") : secret "insights-runtime-extractor-tls" not found Nov 03 19:56:53.078187 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.078115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97d28296-595f-4a77-926d-f6c30841085f-signing-cabundle\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:56:53.078187 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:53.078122 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls podName:731a10e2-8c1d-48c4-a453-1d1d589f6ae2 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:54.078109326 +0000 UTC m=+32.205838397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls") pod "dns-default-5zxlx" (UID: "731a10e2-8c1d-48c4-a453-1d1d589f6ae2") : secret "dns-default-metrics-tls" not found Nov 03 19:56:53.078187 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.078180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czx85\" (UniqueName: \"kubernetes.io/projected/97d28296-595f-4a77-926d-f6c30841085f-kube-api-access-czx85\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:56:53.078363 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.078203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:53.078363 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:53.078264 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:56:53.078363 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:53.078308 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm podName:224062e4-7b08-4d73-ba03-1789a9f591be nodeName:}" failed. No retries permitted until 2026-11-03 19:56:54.078295756 +0000 UTC m=+32.206024822 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6tmzf" (UID: "224062e4-7b08-4d73-ba03-1789a9f591be") : configmap "kube-rbac-proxy" not found Nov 03 19:56:53.078787 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.078770 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97d28296-595f-4a77-926d-f6c30841085f-signing-cabundle\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:56:53.080336 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.080319 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97d28296-595f-4a77-926d-f6c30841085f-signing-key\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:56:53.089871 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.089850 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czx85\" (UniqueName: \"kubernetes.io/projected/97d28296-595f-4a77-926d-f6c30841085f-kube-api-access-czx85\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:56:53.181851 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.181750 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:56:53.354585 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.354550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:53.355001 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.354563 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:53.355001 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.354760 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:53.359375 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.359353 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-72cwb\"" Nov 03 19:56:53.360674 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.360657 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:56:53.360768 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.360681 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:56:53.361127 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.360963 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:56:53.361127 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.360969 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sztcx\"" Nov 03 19:56:53.361127 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.361003 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:56:53.361127 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.361044 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hc4hl\"" Nov 03 19:56:53.478085 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.478016 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-zfvf2"] Nov 03 19:56:53.480839 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:53.480815 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d28296_595f_4a77_926d_f6c30841085f.slice/crio-6ca45d96660899cb5c1d7285486881395ea8230d877ec0c435e219b490fb6d52 WatchSource:0}: Error finding container 6ca45d96660899cb5c1d7285486881395ea8230d877ec0c435e219b490fb6d52: Status 404 returned error can't find the container with id 6ca45d96660899cb5c1d7285486881395ea8230d877ec0c435e219b490fb6d52 Nov 03 19:56:53.641651 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.641615 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" event={"ID":"97d28296-595f-4a77-926d-f6c30841085f","Type":"ContainerStarted","Data":"6ca45d96660899cb5c1d7285486881395ea8230d877ec0c435e219b490fb6d52"} Nov 03 19:56:53.642688 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:53.642665 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxxzr" event={"ID":"ac15ca75-0dd3-4f05-b21d-1ae2280d0415","Type":"ContainerStarted","Data":"82bc135f2f1029b30e6ffb0ff31aeee61cd7290ca6797a6b023ab1b942cf88a0"} Nov 03 19:56:54.085648 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:54.085611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:54.085856 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:54.085708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:54.085856 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:54.085742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:54.085856 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:54.085813 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:56:54.086016 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:54.085886 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls podName:224062e4-7b08-4d73-ba03-1789a9f591be nodeName:}" failed. No retries permitted until 2026-11-03 19:56:56.08586671 +0000 UTC m=+34.213595782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tmzf" (UID: "224062e4-7b08-4d73-ba03-1789a9f591be") : secret "insights-runtime-extractor-tls" not found Nov 03 19:56:54.086016 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:54.085901 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:56:54.086016 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:54.085951 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls podName:731a10e2-8c1d-48c4-a453-1d1d589f6ae2 nodeName:}" failed. No retries permitted until 2026-11-03 19:56:56.085936586 +0000 UTC m=+34.213665664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls") pod "dns-default-5zxlx" (UID: "731a10e2-8c1d-48c4-a453-1d1d589f6ae2") : secret "dns-default-metrics-tls" not found Nov 03 19:56:54.086016 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:54.085994 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:56:54.086225 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:54.086025 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm podName:224062e4-7b08-4d73-ba03-1789a9f591be nodeName:}" failed. No retries permitted until 2026-11-03 19:56:56.086015469 +0000 UTC m=+34.213744545 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6tmzf" (UID: "224062e4-7b08-4d73-ba03-1789a9f591be") : configmap "kube-rbac-proxy" not found Nov 03 19:56:54.710811 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:54.710775 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ssj5l_7f215e51-f5cf-4103-9904-693d5721e647/dns-node-resolver/0.log" Nov 03 19:56:55.096584 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:55.096120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:56:55.096584 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:55.096229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:55.096584 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:55.096266 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qtn\" (UniqueName: \"kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn\") pod \"network-check-target-h4zrs\" (UID: \"c98e82ab-c3b3-4d18-b381-083fdb90b448\") " pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:55.098959 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:55.098872 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 19:56:55.099080 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:55.099012 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs podName:33806d61-5ac2-49ba-9fc9-c89046f2ff03 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:27.098980191 +0000 UTC m=+65.226709270 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs") pod "network-metrics-daemon-st6tw" (UID: "33806d61-5ac2-49ba-9fc9-c89046f2ff03") : secret "metrics-daemon-secret" not found Nov 03 19:56:55.099790 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:55.099765 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qtn\" (UniqueName: \"kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn\") pod \"network-check-target-h4zrs\" (UID: \"c98e82ab-c3b3-4d18-b381-083fdb90b448\") " pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:55.101375 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:55.101352 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:55.180003 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:55.179968 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:56:55.199542 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:55.199517 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:56:55.464299 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:55.464208 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-h4zrs"] Nov 03 19:56:55.468535 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:55.468503 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc98e82ab_c3b3_4d18_b381_083fdb90b448.slice/crio-c34683fcede7837c671bf4d095706ff53599960ffeb4d9b2c3197a59e0bd86ae WatchSource:0}: Error finding container c34683fcede7837c671bf4d095706ff53599960ffeb4d9b2c3197a59e0bd86ae: Status 404 returned error can't find the container with id c34683fcede7837c671bf4d095706ff53599960ffeb4d9b2c3197a59e0bd86ae Nov 03 19:56:55.484736 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:55.484712 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l5wn9"] Nov 03 19:56:55.487962 ip-10-0-132-25 kubenswrapper[2579]: W1103 19:56:55.487875 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6338d09_ef6a_4cf5_8a06_bd466bc5076d.slice/crio-72cd202af312462c65227101bbdd574bc8caf94481e81656e9b628f85990584e WatchSource:0}: Error finding container 72cd202af312462c65227101bbdd574bc8caf94481e81656e9b628f85990584e: Status 404 returned error can't find the container with id 72cd202af312462c65227101bbdd574bc8caf94481e81656e9b628f85990584e Nov 03 19:56:55.649140 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:55.649103 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4zrs" event={"ID":"c98e82ab-c3b3-4d18-b381-083fdb90b448","Type":"ContainerStarted","Data":"c34683fcede7837c671bf4d095706ff53599960ffeb4d9b2c3197a59e0bd86ae"} Nov 03 19:56:55.650337 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:55.650305 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l5wn9" event={"ID":"e6338d09-ef6a-4cf5-8a06-bd466bc5076d","Type":"ContainerStarted","Data":"72cd202af312462c65227101bbdd574bc8caf94481e81656e9b628f85990584e"} Nov 03 19:56:55.709108 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:55.709079 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cwtc9_ffa0c959-6bd1-4feb-b713-3545a6896409/node-ca/0.log" Nov 03 19:56:56.103503 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:56.103073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:56.103503 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:56.103129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:56:56.103503 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:56.103198 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:56:56.103503 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:56.103205 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 19:56:56.103503 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:56.103283 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm podName:224062e4-7b08-4d73-ba03-1789a9f591be nodeName:}" failed. No retries permitted until 2026-11-03 19:57:00.103262306 +0000 UTC m=+38.230991363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6tmzf" (UID: "224062e4-7b08-4d73-ba03-1789a9f591be") : configmap "kube-rbac-proxy" not found Nov 03 19:56:56.103503 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:56.103327 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:56:56.103503 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:56.103377 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls podName:224062e4-7b08-4d73-ba03-1789a9f591be nodeName:}" failed. No retries permitted until 2026-11-03 19:57:00.10336016 +0000 UTC m=+38.231089434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tmzf" (UID: "224062e4-7b08-4d73-ba03-1789a9f591be") : secret "insights-runtime-extractor-tls" not found Nov 03 19:56:56.103503 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:56.103447 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:56:56.103503 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:56:56.103479 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls podName:731a10e2-8c1d-48c4-a453-1d1d589f6ae2 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:00.103468573 +0000 UTC m=+38.231197657 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls") pod "dns-default-5zxlx" (UID: "731a10e2-8c1d-48c4-a453-1d1d589f6ae2") : secret "dns-default-metrics-tls" not found Nov 03 19:56:58.662574 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:58.662532 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" event={"ID":"97d28296-595f-4a77-926d-f6c30841085f","Type":"ContainerStarted","Data":"af6b194495f927ff87e2597d683f46b0889749baf864649856acc7e9b3d6f071"} Nov 03 19:56:58.694455 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:56:58.693517 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" podStartSLOduration=1.9235933649999999 podStartE2EDuration="6.693498998s" totalImagesPullingTime="4.769905633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:56:58.693084307 +0000 UTC m=+36.820813386" watchObservedRunningTime="2026-11-03 19:56:58.693498998 +0000 UTC m=+36.821228079" Nov 03 19:57:00.140559 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:57:00.140526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:00.140887 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:57:00.140573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:00.140887 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:57:00.140626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:00.140887 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:57:00.140728 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:57:00.140887 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:57:00.140784 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls podName:224062e4-7b08-4d73-ba03-1789a9f591be nodeName:}" failed. No retries permitted until 2026-11-03 19:57:08.140764535 +0000 UTC m=+46.268493615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tmzf" (UID: "224062e4-7b08-4d73-ba03-1789a9f591be") : secret "insights-runtime-extractor-tls" not found Nov 03 19:57:00.141015 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:57:00.141005 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:00.141298 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:57:00.141269 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:57:00.141401 ip-10-0-132-25 kubenswrapper[2579]: E1103 19:57:00.141336 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls podName:731a10e2-8c1d-48c4-a453-1d1d589f6ae2 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:08.141316614 +0000 UTC m=+46.269045670 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls") pod "dns-default-5zxlx" (UID: "731a10e2-8c1d-48c4-a453-1d1d589f6ae2") : secret "dns-default-metrics-tls" not found Nov 03 19:57:00.669595 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:57:00.669555 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qxxzr" event={"ID":"ac15ca75-0dd3-4f05-b21d-1ae2280d0415","Type":"ContainerStarted","Data":"84f3eeafdb6904a433cef347b0e717bf98f7179b86b4f79c0583740af7a0b7ff"} Nov 03 19:57:00.671488 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:57:00.671458 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-h4zrs" event={"ID":"c98e82ab-c3b3-4d18-b381-083fdb90b448","Type":"ContainerStarted","Data":"4d582ed8906fc194f3deafdb95ac1e0fcd47b7426cc003b57fbfe3d1dff3af67"} Nov 03 19:57:00.671626 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:57:00.671514 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:57:00.690212 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:57:00.690156 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qxxzr" podStartSLOduration=1.654398372 podStartE2EDuration="8.690142896s" totalImagesPullingTime="7.035744524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:00.689302818 +0000 UTC m=+38.817031899" watchObservedRunningTime="2026-11-03 19:57:00.690142896 +0000 UTC m=+38.817871975" Nov 03 19:57:01.549375 ip-10-0-132-25 kubenswrapper[2579]: I1103 19:57:01.549287 2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:57:01.549503 ip-10-0-132-25 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 19:57:01.567789 ip-10-0-132-25 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 19:57:01.567950 ip-10-0-132-25 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 19:57:01.567998 ip-10-0-132-25 systemd[1]: kubelet.service: Consumed 2.248s CPU time. Nov 03 19:57:01.581053 ip-10-0-132-25 systemd[1]: Starting Kubernetes Kubelet... Nov 03 19:57:01.667889 ip-10-0-132-25 kubenswrapper[4457]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:57:01.667889 ip-10-0-132-25 kubenswrapper[4457]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 19:57:01.667889 ip-10-0-132-25 kubenswrapper[4457]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:57:01.667889 ip-10-0-132-25 kubenswrapper[4457]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 19:57:01.670024 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.669944 4457 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:01.670024 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670023 4457 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:01.670024 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670028 4457 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670031 4457 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670034 4457 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670037 4457 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670040 4457 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670043 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670046 4457 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670065 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670068 4457 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670071 4457 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670076 4457 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670080 4457 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670084 4457 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670088 4457 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670091 4457 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670093 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670096 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670099 4457 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670103 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:01.670116 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670106 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670109 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670111 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670114 4457 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670117 4457 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670120 4457 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670123 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670126 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670129 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670132 4457 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670135 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670138 4457 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670140 4457 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670143 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670145 4457 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670148 4457 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670152 4457 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670154 4457 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670157 4457 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670160 4457 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:01.670619 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670163 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670166 4457 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670168 4457 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670171 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670173 4457 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670176 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670179 4457 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670181 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670184 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670186 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670189 4457 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670191 4457 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670196 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670199 4457 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670201 4457 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670204 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670207 4457 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670210 4457 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670213 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:01.671126 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670215 4457 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670218 4457 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670221 4457 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670223 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670226 4457 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670229 4457 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670231 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670234 4457 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670236 4457 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670239 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670242 4457 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670244 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670247 4457 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670250 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670253 4457 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670256 4457 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670259 4457 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670262 4457 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670265 4457 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670267 4457 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:01.671599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670270 4457 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670272 4457 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670275 4457 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670278 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670280 4457 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670283 4457 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670285 4457 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670288 4457 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670291 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670295 4457 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670299 4457 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670302 4457 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670304 4457 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670307 4457 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670310 4457 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670312 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670315 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670317 4457 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670321 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:01.672071 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670324 4457 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670326 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670329 4457 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670332 4457 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670335 4457 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670337 4457 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.670341 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671121 4457 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671128 4457 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671132 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671136 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671139 4457 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671141 4457 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671144 4457 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671147 4457 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671150 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671153 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671155 4457 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671158 4457 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671160 4457 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:01.672547 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671163 4457 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671166 4457 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671168 4457 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671170 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671173 4457 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671176 4457 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671178 4457 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671181 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671183 4457 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671186 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671188 4457 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671191 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671194 4457 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671197 4457 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671200 4457 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671202 4457 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671205 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671207 4457 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671210 4457 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:01.673033 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671213 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671216 4457 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671218 4457 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671222 4457 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671226 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671230 4457 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671232 4457 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671235 4457 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671237 4457 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671240 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671242 4457 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671245 4457 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671248 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671250 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671253 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671255 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671257 4457 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671260 4457 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671262 4457 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:01.673513 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671265 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671268 4457 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671274 4457 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671277 4457 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671279 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671282 4457 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671285 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671287 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671290 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671292 4457 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671295 4457 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671297 4457 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671299 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671302 4457 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671312 4457 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671315 4457 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671318 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671321 4457 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671323 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:01.674051 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671326 4457 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671328 4457 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671331 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671333 4457 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671337 4457 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671340 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671342 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671345 4457 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671347 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671350 4457 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671352 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671355 4457 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671357 4457 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671360 4457 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671362 4457 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671366 4457 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671371 4457 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671374 4457 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671377 4457 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:01.674580 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671380 4457 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671397 4457 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671400 4457 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671403 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671405 4457 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671408 4457 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671411 4457 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671413 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671416 4457 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671424 4457 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671429 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671431 4457 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671434 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671436 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671438 4457 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671441 4457 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671443 4457 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671945 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671950 4457 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671953 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:01.675077 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671956 4457 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671959 4457 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671961 4457 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671965 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671967 4457 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671970 4457 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671973 4457 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671975 4457 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671981 4457 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671983 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671986 4457 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671988 4457 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671991 4457 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671994 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671996 4457 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.671999 4457 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672002 4457 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672004 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672007 4457 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672010 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:01.675577 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672012 4457 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672020 4457 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672023 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672026 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672029 4457 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672031 4457 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672034 4457 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672037 4457 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672039 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672042 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672044 4457 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672047 4457 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672050 4457 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672052 4457 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672055 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672057 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672060 4457 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672062 4457 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672065 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672067 4457 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:01.676082 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672071 4457 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672074 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672076 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672079 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672082 4457 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672085 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672087 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672090 4457 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672092 4457 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672095 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672097 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672100 4457 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672102 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672106 4457 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672110 4457 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672113 4457 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672116 4457 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672118 4457 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672121 4457 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672123 4457 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:01.676572 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672126 4457 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672133 4457 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672137 4457 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672140 4457 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672143 4457 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672146 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672149 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672151 4457 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672154 4457 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672156 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672159 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672161 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672165 4457 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672168 4457 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672170 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672173 4457 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672176 4457 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672178 4457 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672181 4457 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:01.677034 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672183 4457 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672186 4457 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672188 4457 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672191 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672193 4457 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672196 4457 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672198 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672201 4457 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672204 4457 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672207 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672210 4457 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672212 4457 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672215 4457 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672217 4457 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672221 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672224 4457 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672226 4457 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672229 4457 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672231 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:01.677491 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672234 4457 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672236 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672239 4457 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672241 4457 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.672244 4457 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672348 4457 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672363 4457 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672372 4457 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672378 4457 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672394 4457 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672398 4457 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672402 4457 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672407 4457 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672410 4457 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672413 4457 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672417 4457 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672420 4457 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672423 4457 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672426 4457 flags.go:64] FLAG: --cgroup-root="" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672429 4457 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672432 4457 flags.go:64] FLAG: --client-ca-file="" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672434 4457 flags.go:64] FLAG: --cloud-provider="external" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672438 4457 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672442 4457 flags.go:64] FLAG: --cluster-domain="" Nov 03 19:57:01.677969 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672444 4457 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672447 4457 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672451 4457 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672456 4457 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672460 4457 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672463 4457 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672466 4457 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672469 4457 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672472 4457 flags.go:64] FLAG: --contention-profiling="false" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672475 4457 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672478 4457 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672482 4457 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672484 4457 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672488 4457 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672493 4457 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672496 4457 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672499 4457 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672502 4457 flags.go:64] FLAG: --enable-server="true" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672505 4457 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672512 4457 flags.go:64] FLAG: --event-burst="100" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672515 4457 flags.go:64] FLAG: --event-qps="50" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672518 4457 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672521 4457 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672524 4457 flags.go:64] FLAG: --eviction-hard="" Nov 03 19:57:01.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672528 4457 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672531 4457 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672534 4457 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672537 4457 flags.go:64] FLAG: --eviction-soft="" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672540 4457 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672543 4457 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672546 4457 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672550 4457 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672553 4457 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672556 4457 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672559 4457 flags.go:64] FLAG: --feature-gates="" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672564 4457 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672567 4457 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672571 4457 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672574 4457 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672577 4457 flags.go:64] FLAG: --healthz-port="10248" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672580 4457 flags.go:64] FLAG: --help="false" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672582 4457 flags.go:64] FLAG: --hostname-override="ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672586 4457 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672588 4457 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672591 4457 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672595 4457 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672599 4457 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 19:57:01.679244 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672602 4457 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672604 4457 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672607 4457 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672610 4457 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672613 4457 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672616 4457 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672619 4457 flags.go:64] FLAG: --kube-reserved="" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672622 4457 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672624 4457 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672627 4457 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672630 4457 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672633 4457 flags.go:64] FLAG: --lock-file="" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672635 4457 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672638 4457 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672642 4457 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672647 4457 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672650 4457 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672654 4457 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672657 4457 flags.go:64] FLAG: --logging-format="text" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672659 4457 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672663 4457 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672665 4457 flags.go:64] FLAG: --manifest-url="" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672668 4457 flags.go:64] FLAG: --manifest-url-header="" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672672 4457 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672675 4457 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 19:57:01.679807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672679 4457 flags.go:64] FLAG: --max-pods="110" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672682 4457 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672685 4457 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672688 4457 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672691 4457 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672694 4457 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672697 4457 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672700 4457 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672707 4457 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672710 4457 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672713 4457 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672716 4457 flags.go:64] FLAG: --pod-cidr="" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672719 4457 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672722 4457 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672725 4457 flags.go:64] FLAG: --pods-per-core="0" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672727 4457 flags.go:64] FLAG: --port="10250" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672730 4457 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672733 4457 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-08ee6972ad3dc691c" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672737 4457 flags.go:64] FLAG: --qos-reserved="" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672739 4457 flags.go:64] FLAG: --read-only-port="10255" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672742 4457 flags.go:64] FLAG: --register-node="true" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672745 4457 flags.go:64] FLAG: --register-with-taints="" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672750 4457 flags.go:64] FLAG: --registry-burst="10" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672753 4457 flags.go:64] FLAG: --registry-qps="5" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672755 4457 flags.go:64] FLAG: --reserved-cpus="" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672759 4457 flags.go:64] FLAG: --reserved-memory="" Nov 03 19:57:01.680967 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672762 4457 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672765 4457 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672768 4457 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672771 4457 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672774 4457 flags.go:64] FLAG: --runonce="false" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672777 4457 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672779 4457 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672782 4457 flags.go:64] FLAG: --seccomp-default="false" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672785 4457 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672788 4457 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672791 4457 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672794 4457 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672797 4457 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672801 4457 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672803 4457 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672806 4457 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672809 4457 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672812 4457 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672815 4457 flags.go:64] FLAG: --system-cgroups="" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672818 4457 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672822 4457 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672825 4457 flags.go:64] FLAG: --tls-cert-file="" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672828 4457 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672831 4457 flags.go:64] FLAG: --tls-min-version="" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672834 4457 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 19:57:01.681639 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672837 4457 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672840 4457 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672843 4457 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672846 4457 flags.go:64] FLAG: --v="2" Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672850 4457 flags.go:64] FLAG: --version="false" Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672854 4457 flags.go:64] FLAG: --vmodule="" Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672867 4457 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.672871 4457 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673005 4457 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673009 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673011 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673014 4457 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673017 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673020 4457 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673022 4457 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673025 4457 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673028 4457 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673030 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673035 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673038 4457 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673040 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:01.682319 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673044 4457 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673047 4457 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673049 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673053 4457 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673055 4457 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673058 4457 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673061 4457 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673063 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673067 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673069 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673072 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673075 4457 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673077 4457 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673080 4457 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673082 4457 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673085 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673088 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673090 4457 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673094 4457 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:01.683069 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673100 4457 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673103 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673105 4457 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673108 4457 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673110 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673113 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673115 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673118 4457 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673120 4457 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673123 4457 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673126 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673129 4457 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673132 4457 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673134 4457 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673137 4457 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673139 4457 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673142 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673144 4457 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673147 4457 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673149 4457 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:01.683822 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673152 4457 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673155 4457 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673157 4457 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673160 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673162 4457 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673165 4457 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673168 4457 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673170 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673173 4457 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673175 4457 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673179 4457 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673183 4457 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673187 4457 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673191 4457 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673193 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673196 4457 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673198 4457 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673201 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673203 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673206 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:01.684471 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673209 4457 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673211 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673215 4457 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673217 4457 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673220 4457 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673223 4457 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673225 4457 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673228 4457 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673230 4457 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673233 4457 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673235 4457 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673238 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673240 4457 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673243 4457 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673245 4457 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673248 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673250 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673253 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673255 4457 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:01.685055 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673258 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673261 4457 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673263 4457 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673266 4457 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673268 4457 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673272 4457 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673274 4457 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673278 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673280 4457 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673283 4457 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673285 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673288 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673290 4457 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673293 4457 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673295 4457 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.673322 4457 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:57:01.686005 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673358 4457 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673362 4457 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673365 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673368 4457 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673371 4457 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673375 4457 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673379 4457 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673382 4457 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673398 4457 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673403 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673407 4457 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673412 4457 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673415 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673418 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673420 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673423 4457 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673425 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673428 4457 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673431 4457 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673433 4457 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:01.686486 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673436 4457 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673440 4457 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673442 4457 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673445 4457 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673448 4457 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673451 4457 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673454 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673456 4457 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673459 4457 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673461 4457 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673464 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673467 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673471 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673473 4457 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673476 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673478 4457 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673481 4457 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673484 4457 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673487 4457 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:01.687130 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673489 4457 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673492 4457 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673495 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673497 4457 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673500 4457 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673502 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673505 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673508 4457 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673510 4457 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673513 4457 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673515 4457 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673518 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673521 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673523 4457 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673528 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673530 4457 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673533 4457 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673536 4457 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673539 4457 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:01.687679 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673541 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673544 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673547 4457 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673549 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673552 4457 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673555 4457 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673559 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673562 4457 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673565 4457 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673567 4457 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673570 4457 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673573 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673575 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673578 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673580 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673583 4457 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673586 4457 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673588 4457 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673591 4457 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673593 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:01.688134 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673596 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673599 4457 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673602 4457 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673606 4457 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673609 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673612 4457 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673615 4457 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673618 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673621 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673624 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673626 4457 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673629 4457 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673632 4457 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673635 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673637 4457 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673640 4457 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673642 4457 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673645 4457 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673648 4457 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:01.688625 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673651 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673653 4457 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673656 4457 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673659 4457 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673662 4457 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673665 4457 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673668 4457 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673670 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.673673 4457 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.677270 4457 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.677285 4457 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.677304 4457 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.677308 4457 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677363 4457 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677369 4457 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677373 4457 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677377 4457 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677380 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677397 4457 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:01.689070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677402 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677407 4457 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677411 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677414 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677417 4457 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677419 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677422 4457 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677425 4457 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677427 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677430 4457 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677433 4457 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677436 4457 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677439 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677441 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677444 4457 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677447 4457 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677449 4457 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677452 4457 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677454 4457 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677457 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:01.689592 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677460 4457 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677462 4457 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677465 4457 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677468 4457 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677471 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677474 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677476 4457 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677479 4457 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677481 4457 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677484 4457 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677487 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677490 4457 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677493 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677497 4457 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677502 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677505 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677508 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677510 4457 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677513 4457 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:01.690107 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677516 4457 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677518 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677521 4457 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677524 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677526 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677529 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677532 4457 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677534 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677537 4457 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677540 4457 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677542 4457 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677545 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677547 4457 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677550 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677552 4457 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677555 4457 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677557 4457 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677560 4457 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677563 4457 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677566 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:01.690615 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677568 4457 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677571 4457 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677574 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677576 4457 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677579 4457 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677581 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677583 4457 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677586 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677588 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677591 4457 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677593 4457 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677596 4457 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677598 4457 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677601 4457 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677603 4457 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677606 4457 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677609 4457 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677611 4457 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677614 4457 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677616 4457 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:01.691111 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677619 4457 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677621 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677624 4457 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677626 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677629 4457 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677631 4457 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677634 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677637 4457 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677639 4457 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677642 4457 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677645 4457 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677648 4457 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677651 4457 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677653 4457 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677656 4457 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677658 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677661 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677664 4457 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677666 4457 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677668 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:01.691599 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677671 4457 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.677698 4457 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677866 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677871 4457 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677874 4457 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677877 4457 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677879 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677882 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677884 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677887 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677891 4457 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677895 4457 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677898 4457 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677901 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677904 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677907 4457 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:01.692062 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677910 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677915 4457 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677918 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677920 4457 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677923 4457 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677925 4457 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677929 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677932 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677935 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677938 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677940 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677943 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677945 4457 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677948 4457 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677951 4457 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677953 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677956 4457 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677959 4457 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677961 4457 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:01.692455 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677965 4457 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677968 4457 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677971 4457 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677974 4457 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677976 4457 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677979 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677982 4457 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677985 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677988 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677990 4457 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677993 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677995 4457 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.677998 4457 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678001 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678005 4457 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678008 4457 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678010 4457 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678013 4457 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678015 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678018 4457 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:01.692900 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678020 4457 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678024 4457 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678026 4457 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678029 4457 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678031 4457 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678034 4457 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678037 4457 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678040 4457 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678042 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678045 4457 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678047 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678050 4457 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678054 4457 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678057 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678060 4457 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678063 4457 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678065 4457 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678068 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678071 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678073 4457 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:01.693364 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678076 4457 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678079 4457 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678081 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678084 4457 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678086 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678089 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678093 4457 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678096 4457 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678099 4457 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678101 4457 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678104 4457 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678106 4457 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678109 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678111 4457 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678114 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678117 4457 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678120 4457 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678122 4457 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678125 4457 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:01.693879 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678128 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678132 4457 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678135 4457 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678138 4457 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678141 4457 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678145 4457 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678149 4457 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678152 4457 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678155 4457 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678158 4457 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678161 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678163 4457 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678166 4457 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678169 4457 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.678192 4457 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678222 4457 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 19:57:01.694333 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678226 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678229 4457 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678232 4457 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678236 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678238 4457 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678241 4457 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678244 4457 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678247 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678249 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678252 4457 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678255 4457 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678258 4457 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678261 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678264 4457 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678267 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678269 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678272 4457 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678274 4457 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678277 4457 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 19:57:01.694749 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678279 4457 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678282 4457 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678285 4457 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678288 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678291 4457 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678294 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678296 4457 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678299 4457 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678301 4457 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678304 4457 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678307 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678311 4457 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678314 4457 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678317 4457 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678320 4457 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678323 4457 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678327 4457 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678330 4457 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678332 4457 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 19:57:01.695223 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678335 4457 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678338 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678341 4457 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678344 4457 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678346 4457 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678349 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678351 4457 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678354 4457 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678357 4457 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678359 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678362 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678364 4457 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678367 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678371 4457 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678374 4457 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678377 4457 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678381 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678403 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678406 4457 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 19:57:01.695756 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678409 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678411 4457 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678414 4457 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678416 4457 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678419 4457 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678422 4457 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678425 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678428 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678431 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678433 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678437 4457 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678440 4457 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678443 4457 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678446 4457 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678448 4457 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678451 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678454 4457 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678457 4457 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678459 4457 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 19:57:01.696403 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678462 4457 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678464 4457 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678468 4457 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678471 4457 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678473 4457 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678476 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678479 4457 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678481 4457 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678484 4457 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678486 4457 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678490 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678493 4457 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678496 4457 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678498 4457 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678501 4457 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678503 4457 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678506 4457 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678508 4457 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678511 4457 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678514 4457 feature_gate.go:386] unrecognized feature gate: Example Nov 03 19:57:01.697090 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678516 4457 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678519 4457 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678522 4457 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678527 4457 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678529 4457 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678532 4457 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678535 4457 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678537 4457 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:01.678540 4457 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.678596 4457 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.679531 4457 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.679575 4457 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.680440 4457 server.go:1014] "Starting client certificate rotation" Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.680658 4457 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.680720 4457 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 19:51:22 +0000 UTC" deadline="2028-05-28 04:23:10.131672651 +0000 UTC" Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.680768 4457 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13712h26m8.450907394s" Nov 03 19:57:01.697986 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.681497 4457 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:57:01.698615 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.682703 4457 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 19:57:01.698615 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.684358 4457 log.go:25] "Validated CRI v1 runtime API" Nov 03 19:57:01.698615 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.686249 4457 log.go:25] "Validated CRI v1 image API" Nov 03 19:57:01.698615 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.686986 4457 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 19:57:01.698615 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.688434 4457 fs.go:135] Filesystem UUIDs: map[581dc9db-1ea1-4a6b-a1a5-5aee7ed5d764:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f0a35fe7-61fb-4ae0-bb00-00e3feabca72:/dev/nvme0n1p4] Nov 03 19:57:01.698907 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.688453 4457 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a77dd79af69bc379beceb5011ac869c45b9e91bd7cfda7b3bfcca7b49b2e278/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a77dd79af69bc379beceb5011ac869c45b9e91bd7cfda7b3bfcca7b49b2e278/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35150edf568d96bc082d83b80c1db26cdb6a69c1259c86011542b4cc61f141a6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35150edf568d96bc082d83b80c1db26cdb6a69c1259c86011542b4cc61f141a6/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b0347a2efe0acd92df23648fa9c9d6f64d992dac5e32646c46e123b2239f207/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b0347a2efe0acd92df23648fa9c9d6f64d992dac5e32646c46e123b2239f207/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51fd09239009d0c5bdd7f1f9ec686fb72c32f8abaf5538ca38165bf79a7ef043/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51fd09239009d0c5bdd7f1f9ec686fb72c32f8abaf5538ca38165bf79a7ef043/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ca45d96660899cb5c1d7285486881395ea8230d877ec0c435e219b490fb6d52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ca45d96660899cb5c1d7285486881395ea8230d877ec0c435e219b490fb6d52/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e334e0f5c5cb92cfdf99467adb29e5e8c2c6f51a68919dc8164e62ff7063339/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e334e0f5c5cb92cfdf99467adb29e5e8c2c6f51a68919dc8164e62ff7063339/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72cd202af312462c65227101bbdd574bc8caf94481e81656e9b628f85990584e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72cd202af312462c65227101bbdd574bc8caf94481e81656e9b628f85990584e/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7bc7577922eb0414c657d916675bb16b35a5ad6a22526abf21ba7aae8fc9775e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7bc7577922eb0414c657d916675bb16b35a5ad6a22526abf21ba7aae8fc9775e/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82bc135f2f1029b30e6ffb0ff31aeee61cd7290ca6797a6b023ab1b942cf88a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82bc135f2f1029b30e6ffb0ff31aeee61cd7290ca6797a6b023ab1b942cf88a0/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/830e932e628998416cad3c6aa6444beb61b0ff8a2bad9f109e99ce1b1b23b19d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/830e932e628998416cad3c6aa6444beb61b0ff8a2bad9f109e99ce1b1b23b19d/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92b192f6ef7e7367637dfb3e494ac28951e607715d76272d1eb7024ef0e27aea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92b192f6ef7e7367637dfb3e494ac28951e607715d76272d1eb7024ef0e27aea/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd6773b2f462d346511693cad2335a5ba671730d1886dde1e05ec240083a3169/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd6773b2f462d346511693cad2335a5ba671730d1886dde1e05ec240083a3169/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c34683fcede7837c671bf4d095706ff53599960ffeb4d9b2c3197a59e0bd86ae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c34683fcede7837c671bf4d095706ff53599960ffeb4d9b2c3197a59e0bd86ae/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7e7902ba9bf3721dba8987ea143d83d5910a55d479f46a8fe049347e5508fb4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7e7902ba9bf3721dba8987ea143d83d5910a55d479f46a8fe049347e5508fb4/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/224062e4-7b08-4d73-ba03-1789a9f591be/volumes/kubernetes.io~projected/kube-api-access-nvxvg:{mountpoint:/var/lib/kubelet/pods/224062e4-7b08-4d73-ba03-1789a9f591be/volumes/kubernetes.io~projected/kube-api-access-nvxvg major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2590e213-dc39-406a-8094-b4e01162e221/volumes/kubernetes.io~projected/kube-api-access-8kwj4:{mountpoint:/var/lib/kubelet/pods/2590e213-dc39-406a-8094-b4e01162e221/volumes/kubernetes.io~projected/kube-api-access-8kwj4 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/308feccf-7178-48c7-b370-3c78f96e8919/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/308feccf-7178-48c7-b370-3c78f96e8919/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/308feccf-7178-48c7-b370-3c78f96e8919/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/308feccf-7178-48c7-b370-3c78f96e8919/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/308feccf-7178-48c7-b370-3c78f96e8919/volumes/kubernetes.io~projected/kube-api-access-lplg9:{mountpoint:/var/lib/kubelet/pods/308feccf-7178-48c7-b370-3c78f96e8919/volumes/kubernetes.io~projected/kube-api-access-lplg9 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33806d61-5ac2-49ba-9fc9-c89046f2ff03/volumes/kubernetes.io~projected/kube-api-access-tn92t:{mountpoint:/var/lib/kubelet/pods/33806d61-5ac2-49ba-9fc9-c89046f2ff03/volumes/kubernetes.io~projected/kube-api-access-tn92t major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/642aaac2-4dc2-4800-bc55-c70730884400/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/642aaac2-4dc2-4800-bc55-c70730884400/volumes/kubernetes.io~secret/agent-certs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/731a10e2-8c1d-48c4-a453-1d1d589f6ae2/volumes/kubernetes.io~projected/kube-api-access-dqljf:{mountpoint:/var/lib/kubelet/pods/731a10e2-8c1d-48c4-a453-1d1d589f6ae2/volumes/kubernetes.io~projected/kube-api-access-dqljf major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f215e51-f5cf-4103-9904-693d5721e647/volumes/kubernetes.io~projected/kube-api-access-mgbht:{mountpoint:/var/lib/kubelet/pods/7f215e51-f5cf-4103-9904-693d5721e647/volumes/kubernetes.io~projected/kube-api-access-mgbht major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97d28296-595f-4a77-926d-f6c30841085f/volumes/kubernetes.io~projected/kube-api-access-czx85:{mountpoint:/var/lib/kubelet/pods/97d28296-595f-4a77-926d-f6c30841085f/volumes/kubernetes.io~projected/kube-api-access-czx85 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97d28296-595f-4a77-926d-f6c30841085f/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/97d28296-595f-4a77-926d-f6c30841085f/volumes/kubernetes.io~secret/signing-key major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac15ca75-0dd3-4f05-b21d-1ae2280d0415/volumes/kubernetes.io~projected/kube-api-access-qm4bm:{mountpoint:/var/lib/kubelet/pods/ac15ca75-0dd3-4f05-b21d-1ae2280d0415/volumes/kubernetes.io~projected/kube-api-access-qm4bm major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ac15ca75-0dd3-4f05-b21d-1ae2280d0415/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ac15ca75-0dd3-4f05-b21d-1ae2280d0415/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af954863-9eb8-4bd6-a64b-918f5f4ff6b0/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/af954863-9eb8-4bd6-a64b-918f5f4ff6b0/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af954863-9eb8-4bd6-a64b-918f5f4ff6b0/volumes/kubernetes.io~projected/kube-api-access-wz2q2:{mountpoint:/var/lib/kubelet/pods/af954863-9eb8-4bd6-a64b-918f5f4ff6b0/volumes/kubernetes.io~projected/kube-api-access-wz2q2 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af954863-9eb8-4bd6-a64b-918f5f4ff6b0/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/af954863-9eb8-4bd6-a64b-918f5f4ff6b0/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b33b71b3-287e-4516-9d0a-726f302a8dff/volumes/kubernetes.io~projected/kube-api-access-8qspj:{mountpoint:/var/lib/kubelet/pods/b33b71b3-287e-4516-9d0a-726f302a8dff/volumes/kubernetes.io~projected/kube-api-access-8qspj major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2c45b0c-9bd5-42bc-8a11-b2db15876d63/volumes/kubernetes.io~projected/kube-api-access-7jw95:{mountpoint:/var/lib/kubelet/pods/c2c45b0c-9bd5-42bc-8a11-b2db15876d63/volumes/kubernetes.io~projected/kube-api-access-7jw95 major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c98e82ab-c3b3-4d18-b381-083fdb90b448/volumes/kubernetes.io~projected/kube-api-access-75qtn:{mountpoint:/var/lib/kubelet/pods/c98e82ab-c3b3-4d18-b381-083fdb90b448/volumes/kubernetes.io~projected/kube-api-access-75qtn major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9/volumes/kubernetes.io~projected/kube-api-access-fp952:{mountpoint:/var/lib/kubelet/pods/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9/volumes/kubernetes.io~projected/kube-api-access-fp952 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6338d09-ef6a-4cf5-8a06-bd466bc5076d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/e6338d09-ef6a-4cf5-8a06-bd466bc5076d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ffa0c959-6bd1-4feb-b713-3545a6896409/volumes/kubernetes.io~projected/kube-api-access-hggp5:{mountpoint:/var/lib/kubelet/pods/ffa0c959-6bd1-4feb-b713-3545a6896409/volumes/kubernetes.io~projected/kube-api-access-hggp5 major:0 minor:67 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3ad4a28073cf8d99e53f287787e0231b2f6b86c85aa75af91e53a0b9092921bc/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/2e7890f3229c5793cdbfcb969b91fa9ecb8da68c7755e29a25e2efc35e5e0394/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4d24d1c6363eee6b5ee69a85c83f5d7d8281e6a960e76fdde983186f19f1088a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/77aedfa80251b411c1286530774e8526c412d93097a5fef35e52dc6e2baeb81c/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-115:{mountpoint:/var/lib/containers/storage/overlay/5d0bcd8fd71c13e87a7fb3df23aa148c631981d006ad6c45985aae284a5a35da/merged major:0 minor:115 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/317141d6b2f9831a9b0bda7e767ee66db5fe44dc934cc2852079a0ba314ae2cc/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/90dfb3fa50ad75cd1383f9be5823ff0a7ce41fb11452f13fb207f4b407861638/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/1e0292ef505d309b923347bc7860b59c5f18b56c250fb1ae2f5ca8568a5b8450/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-128:{mountpoint:/var/lib/containers/storage/overlay/eef7a7e50bd01905c046fbe639ed8c1482e5f3525413cab65b59e886ae46fc0c/merged major:0 minor:128 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/99dd2df5afb4fe079d5d669375964c2dfd7916bd077a105c30a5548746e42eca/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/59421419c091583690b6b260a8057a49e0f064b221c711a2e759d98bcc9511ff/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/254d28a636125185d974c2d5cac7ec10c57674f7c5872cd14c4b200a1f2cb941/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/097d5d04c94954e3f20439942e4fceddb398a5c0b4ba2e942477113e1f774a0e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/abbeb6701e4cd24e61d647140ef0013040782c41f7557edd8159c58ffc430c48/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/023b62f71e7e9d2537e12cbd0e907d3d97b0e6b58c23717a5f92308d2145b999/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1c24e54b82961cd55a2c691d48b77032579bf033c6511e3e2d6371e906665048/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/44deadcc68f684589a0f7553d550d1e1820cac88fc9003468889ff38255915a1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a63a9f02d0bf8cafae6e2efae41f4fc02ae89212c06c4ffdcf407733e3116d2c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/543467f4dca4efa41c1a563b39219b45cca4ac93b3a8dc36f8ce46cf7e75cbaf/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e6da319afabe8b721b89251fa9cc686f1b05e0c39e8cff41c962390d4030bcc9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/a6c1799b41a7020273c4d58c8f77576759ac30a940e554756180ea6d5a00910b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/ad2499e5e4d957625a2e1dbd029379574593c259447640690314a571a4036d35/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/10959d0b61744c0f2c9f6d1b21c54d5711db378ad4d8949b7d3e590c4f1cf272/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/cea34cd75284323c5ae30e05f043169a3e4c1e0d4048f593a748e999d1e41ed4/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/9b2024d1d679f7c5eef9eb39e41ac49a02800075810220cd9a3e69e87155b21f/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/1cf5577e06bf4167c8483c8681e36d7282ed2d23c51e9929413a14d63c6540b1/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/3bd3102b5282101a9b2a3458dc6dfe188643f5b6e2584e19fb3a07542873d1dc/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/09294054a51610aff86b85d184a6f6e8ead53103a9816da9275d6665cada9331/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/e03cd424aa3cb153abfbd3e213dedc69b197760068d8a61f1c028c2024ac9f19/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-52:{mountpoint:/var/lib/containers/storage/overlay/f926e159bb11a3eddefe1df34f9b7041cf5d2fca914d13bdc4a7f1a17e77f035/merged major:0 minor:52 fsType:overlay blockSize:0} overlay_0-54:{mountpoint:/var/lib/containers/storage/overlay/10be8e87d2d6b2477edf8f319b0f5144607c97b8fbcc599e09eebcc666e3fc5f/merged major:0 minor:54 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/4417ba90da7c7fdae403511527bf087365c41dbc894deba29159b0eff2532162/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/b103ab6b3e4607067280ff40902e82e49aafbb9e72df099e2d41b585549ce14d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3b75af00748041f34447e251541d5cd9d65c1bff8edd9d6805f01252c313ad80/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/8bfc63b142c04d0366a4a67b1a23291806e046273a92bc5f8bb69235920d0611/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/218cdc316541ad5c8be4a378334f592c8612d7f639a817d4e54c85969185781b/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6915fb777f051c0fc4cc173f8818f2f1810f71c357ee818693eb52b18cd7e1c3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e832a4f731f80c508e03286022cef23ac0db01bdefb30485da4568d1f01c3e60/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ce18d5a7cce856809ffddc48d3da736d8a795d6e1863ec3bd895ed4b19b7c416/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.694336 4457 manager.go:217] Machine: {Timestamp:2026-11-03 19:57:01.693888498 +0000 UTC m=+0.069485235 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3180195 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f64289fc145b566f89e0965145e68 SystemUUID:ec2f6428-9fc1-45b5-66f8-9e0965145e68 BootID:92a6cca9-cfaa-45bc-9cab-ad785243a6ec Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd6773b2f462d346511693cad2335a5ba671730d1886dde1e05ec240083a3169/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7bc7577922eb0414c657d916675bb16b35a5ad6a22526abf21ba7aae8fc9775e/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac15ca75-0dd3-4f05-b21d-1ae2280d0415/volumes/kubernetes.io~projected/kube-api-access-qm4bm DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51fd09239009d0c5bdd7f1f9ec686fb72c32f8abaf5538ca38165bf79a7ef043/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7e7902ba9bf3721dba8987ea143d83d5910a55d479f46a8fe049347e5508fb4/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ac15ca75-0dd3-4f05-b21d-1ae2280d0415/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/731a10e2-8c1d-48c4-a453-1d1d589f6ae2/volumes/kubernetes.io~projected/kube-api-access-dqljf DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c98e82ab-c3b3-4d18-b381-083fdb90b448/volumes/kubernetes.io~projected/kube-api-access-75qtn DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/830e932e628998416cad3c6aa6444beb61b0ff8a2bad9f109e99ce1b1b23b19d/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c2c45b0c-9bd5-42bc-8a11-b2db15876d63/volumes/kubernetes.io~projected/kube-api-access-7jw95 DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/308feccf-7178-48c7-b370-3c78f96e8919/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b0347a2efe0acd92df23648fa9c9d6f64d992dac5e32646c46e123b2239f207/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-115 DeviceMajor:0 DeviceMinor:115 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c34683fcede7837c671bf4d095706ff53599960ffeb4d9b2c3197a59e0bd86ae/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7f215e51-f5cf-4103-9904-693d5721e647/volumes/kubernetes.io~projected/kube-api-access-mgbht DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/308feccf-7178-48c7-b370-3c78f96e8919/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82bc135f2f1029b30e6ffb0ff31aeee61cd7290ca6797a6b023ab1b942cf88a0/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ca45d96660899cb5c1d7285486881395ea8230d877ec0c435e219b490fb6d52/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2590e213-dc39-406a-8094-b4e01162e221/volumes/kubernetes.io~projected/kube-api-access-8kwj4 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35150edf568d96bc082d83b80c1db26cdb6a69c1259c86011542b4cc61f141a6/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/642aaac2-4dc2-4800-bc55-c70730884400/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/224062e4-7b08-4d73-ba03-1789a9f591be/volumes/kubernetes.io~projected/kube-api-access-nvxvg DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72cd202af312462c65227101bbdd574bc8caf94481e81656e9b628f85990584e/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a77dd79af69bc379beceb5011ac869c45b9e91bd7cfda7b3bfcca7b49b2e278/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97d28296-595f-4a77-926d-f6c30841085f/volumes/kubernetes.io~projected/kube-api-access-czx85 DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-128 DeviceMajor:0 DeviceMinor:128 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-54 DeviceMajor:0 DeviceMinor:54 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92b192f6ef7e7367637dfb3e494ac28951e607715d76272d1eb7024ef0e27aea/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e6338d09-ef6a-4cf5-8a06-bd466bc5076d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/af954863-9eb8-4bd6-a64b-918f5f4ff6b0/volumes/kubernetes.io~projected/kube-api-access-wz2q2 DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b33b71b3-287e-4516-9d0a-726f302a8dff/volumes/kubernetes.io~projected/kube-api-access-8qspj DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/33806d61-5ac2-49ba-9fc9-c89046f2ff03/volumes/kubernetes.io~projected/kube-api-access-tn92t DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/97d28296-595f-4a77-926d-f6c30841085f/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9/volumes/kubernetes.io~projected/kube-api-access-fp952 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/ffa0c959-6bd1-4feb-b713-3545a6896409/volumes/kubernetes.io~projected/kube-api-access-hggp5 DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/308feccf-7178-48c7-b370-3c78f96e8919/volumes/kubernetes.io~projected/kube-api-access-lplg9 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af954863-9eb8-4bd6-a64b-918f5f4ff6b0/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-52 DeviceMajor:0 DeviceMinor:52 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af954863-9eb8-4bd6-a64b-918f5f4ff6b0/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e334e0f5c5cb92cfdf99467adb29e5e8c2c6f51a68919dc8164e62ff7063339/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:6ca45d96660899c MacAddress:f2:84:00:58:b0:a3 Speed:10000 Mtu:8901} {Name:72cd202af312462 MacAddress:42:a4:5e:af:a4:92 Speed:10000 Mtu:8901} {Name:82bc135f2f1029b MacAddress:c2:3c:c4:8e:0b:aa Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d0:63:95:7b Speed:0 Mtu:9001} {Name:br-int MacAddress:06:7c:1b:2f:48:75 Speed:0 Mtu:8901} {Name:c34683fcede7837 MacAddress:26:e1:05:5d:e3:09 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d0:63:95:7b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:92:fa:02:ce:2a:7d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a2:9d:a2:e8:f9:20 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.694588 4457 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.694631 4457 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.694919 4457 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.694940 4457 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-25.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.695080 4457 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.695088 4457 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.695098 4457 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.695116 4457 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.695205 4457 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.695358 4457 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.695468 4457 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.695504 4457 kubelet.go:493] "Attempting to sync node with API server" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.695516 4457 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.695526 4457 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.695536 4457 kubelet.go:405] "Adding apiserver pod source" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.695544 4457 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.696199 4457 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.696211 4457 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:57:01.699153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.698272 4457 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 19:57:01.700911 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.700893 4457 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 19:57:01.700911 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.700908 4457 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 19:57:01.701036 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701026 4457 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 19:57:01.701269 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701253 4457 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 19:57:01.701330 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701292 4457 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 19:57:01.701428 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701417 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 19:57:01.701491 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701436 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 19:57:01.701491 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701447 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 19:57:01.701491 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701457 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 19:57:01.701491 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701466 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 19:57:01.701491 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701475 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 19:57:01.701491 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701484 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 19:57:01.701797 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701494 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 19:57:01.701797 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701506 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 19:57:01.701797 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701516 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 19:57:01.701797 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701529 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 19:57:01.701797 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701543 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 19:57:01.701797 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701576 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 19:57:01.701797 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.701585 4457 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 19:57:01.705007 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.704991 4457 server.go:1263] "Started kubelet" Nov 03 19:57:01.705097 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.705070 4457 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 19:57:01.705183 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.705134 4457 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 19:57:01.705363 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.705211 4457 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 19:57:01.705520 ip-10-0-132-25 systemd[1]: Started Kubernetes Kubelet. Nov 03 19:57:01.705675 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.705512 4457 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 19:57:01.706153 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.706138 4457 server.go:317] "Adding debug handlers to kubelet server" Nov 03 19:57:01.707440 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.707420 4457 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 19:57:01.707615 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.707601 4457 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 19:57:01.708242 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:01.708219 4457 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-25.us-east-2.compute.internal\" not found" Nov 03 19:57:01.708330 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.708266 4457 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 19:57:01.708330 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.708273 4457 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 19:57:01.708576 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.708562 4457 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 19:57:01.709567 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.709541 4457 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 19:51:23 +0000 UTC" deadline="2028-04-06 20:43:37.199384982 +0000 UTC" Nov 03 19:57:01.709567 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.709562 4457 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12480h46m35.489824712s" Nov 03 19:57:01.710713 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710628 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="308feccf-7178-48c7-b370-3c78f96e8919" volumeName="kubernetes.io/projected/308feccf-7178-48c7-b370-3c78f96e8919-kube-api-access-lplg9" seLinuxMountContext="" Nov 03 19:57:01.710799 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710718 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731a10e2-8c1d-48c4-a453-1d1d589f6ae2" volumeName="kubernetes.io/configmap/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-config-volume" seLinuxMountContext="" Nov 03 19:57:01.710799 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710767 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af954863-9eb8-4bd6-a64b-918f5f4ff6b0" volumeName="kubernetes.io/secret/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710808 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b33b71b3-287e-4516-9d0a-726f302a8dff" volumeName="kubernetes.io/configmap/b33b71b3-287e-4516-9d0a-726f302a8dff-iptables-alerter-script" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710821 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b33b71b3-287e-4516-9d0a-726f302a8dff" volumeName="kubernetes.io/projected/b33b71b3-287e-4516-9d0a-726f302a8dff-kube-api-access-8qspj" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710834 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33806d61-5ac2-49ba-9fc9-c89046f2ff03" volumeName="kubernetes.io/projected/33806d61-5ac2-49ba-9fc9-c89046f2ff03-kube-api-access-tn92t" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710855 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac15ca75-0dd3-4f05-b21d-1ae2280d0415" volumeName="kubernetes.io/secret/ac15ca75-0dd3-4f05-b21d-1ae2280d0415-cert" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710867 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc075c5d-3e03-4e5c-be0d-09efa41fcdb9" volumeName="kubernetes.io/projected/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-kube-api-access-fp952" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710882 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2590e213-dc39-406a-8094-b4e01162e221" volumeName="kubernetes.io/configmap/2590e213-dc39-406a-8094-b4e01162e221-multus-daemon-config" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710903 4457 factory.go:55] Registering systemd factory Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710915 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2590e213-dc39-406a-8094-b4e01162e221" volumeName="kubernetes.io/projected/2590e213-dc39-406a-8094-b4e01162e221-kube-api-access-8kwj4" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710923 4457 factory.go:223] Registration of the systemd container factory successfully Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710954 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731a10e2-8c1d-48c4-a453-1d1d589f6ae2" volumeName="kubernetes.io/empty-dir/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-tmp-dir" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.710979 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ac15ca75-0dd3-4f05-b21d-1ae2280d0415" volumeName="kubernetes.io/projected/ac15ca75-0dd3-4f05-b21d-1ae2280d0415-kube-api-access-qm4bm" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711002 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa0c959-6bd1-4feb-b713-3545a6896409" volumeName="kubernetes.io/configmap/ffa0c959-6bd1-4feb-b713-3545a6896409-serviceca" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711014 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af954863-9eb8-4bd6-a64b-918f5f4ff6b0" volumeName="kubernetes.io/projected/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-kube-api-access-wz2q2" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711029 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="224062e4-7b08-4d73-ba03-1789a9f591be" volumeName="kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711060 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2590e213-dc39-406a-8094-b4e01162e221" volumeName="kubernetes.io/configmap/2590e213-dc39-406a-8094-b4e01162e221-cni-binary-copy" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711073 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="642aaac2-4dc2-4800-bc55-c70730884400" volumeName="kubernetes.io/configmap/642aaac2-4dc2-4800-bc55-c70730884400-konnectivity-ca" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711112 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731a10e2-8c1d-48c4-a453-1d1d589f6ae2" volumeName="kubernetes.io/projected/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-kube-api-access-dqljf" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711135 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f215e51-f5cf-4103-9904-693d5721e647" volumeName="kubernetes.io/empty-dir/7f215e51-f5cf-4103-9904-693d5721e647-tmp-dir" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711149 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f215e51-f5cf-4103-9904-693d5721e647" volumeName="kubernetes.io/projected/7f215e51-f5cf-4103-9904-693d5721e647-kube-api-access-mgbht" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711165 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97d28296-595f-4a77-926d-f6c30841085f" volumeName="kubernetes.io/configmap/97d28296-595f-4a77-926d-f6c30841085f-signing-cabundle" seLinuxMountContext="" Nov 03 19:57:01.711193 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711190 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="224062e4-7b08-4d73-ba03-1789a9f591be" volumeName="kubernetes.io/empty-dir/224062e4-7b08-4d73-ba03-1789a9f591be-data-volume" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711206 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="308feccf-7178-48c7-b370-3c78f96e8919" volumeName="kubernetes.io/empty-dir/308feccf-7178-48c7-b370-3c78f96e8919-tmp" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711240 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="642aaac2-4dc2-4800-bc55-c70730884400" volumeName="kubernetes.io/secret/642aaac2-4dc2-4800-bc55-c70730884400-agent-certs" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711280 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af954863-9eb8-4bd6-a64b-918f5f4ff6b0" volumeName="kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-env-overrides" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711304 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af954863-9eb8-4bd6-a64b-918f5f4ff6b0" volumeName="kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovnkube-script-lib" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711318 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" volumeName="kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711343 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" volumeName="kubernetes.io/projected/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-kube-api-access-7jw95" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711398 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6338d09-ef6a-4cf5-8a06-bd466bc5076d" volumeName="kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711411 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="224062e4-7b08-4d73-ba03-1789a9f591be" volumeName="kubernetes.io/projected/224062e4-7b08-4d73-ba03-1789a9f591be-kube-api-access-nvxvg" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711439 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97d28296-595f-4a77-926d-f6c30841085f" volumeName="kubernetes.io/projected/97d28296-595f-4a77-926d-f6c30841085f-kube-api-access-czx85" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711454 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="308feccf-7178-48c7-b370-3c78f96e8919" volumeName="kubernetes.io/empty-dir/308feccf-7178-48c7-b370-3c78f96e8919-etc-tuned" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711468 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97d28296-595f-4a77-926d-f6c30841085f" volumeName="kubernetes.io/secret/97d28296-595f-4a77-926d-f6c30841085f-signing-key" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711509 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" volumeName="kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cni-binary-copy" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711526 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" volumeName="kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711555 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ffa0c959-6bd1-4feb-b713-3545a6896409" volumeName="kubernetes.io/projected/ffa0c959-6bd1-4feb-b713-3545a6896409-kube-api-access-hggp5" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711577 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af954863-9eb8-4bd6-a64b-918f5f4ff6b0" volumeName="kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovnkube-config" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711589 4457 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c98e82ab-c3b3-4d18-b381-083fdb90b448" volumeName="kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn" seLinuxMountContext="" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711621 4457 reconstruct.go:97] "Volume reconstruction finished" Nov 03 19:57:01.711939 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.711650 4457 reconciler.go:29] "Reconciler: start to sync state" Nov 03 19:57:01.713871 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.713835 4457 factory.go:153] Registering CRI-O factory Nov 03 19:57:01.713871 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.713854 4457 factory.go:223] Registration of the crio container factory successfully Nov 03 19:57:01.714008 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.713936 4457 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 19:57:01.714361 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.714350 4457 factory.go:103] Registering Raw factory Nov 03 19:57:01.714438 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.714366 4457 manager.go:1199] Started watching for new ooms in manager Nov 03 19:57:01.714742 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.714726 4457 manager.go:319] Starting recovery of all containers Nov 03 19:57:01.721456 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.719840 4457 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:57:01.721564 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.721515 4457 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 19:57:01.728233 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.728093 4457 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:57:01.728928 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.728775 4457 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:57:01.732859 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.732839 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 19:57:01.733872 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.733855 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 19:57:01.747966 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.747933 4457 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 19:57:01.749179 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.749154 4457 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 19:57:01.749179 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.749176 4457 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 19:57:01.749294 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.749196 4457 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 19:57:01.749294 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:01.749234 4457 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 19:57:01.753149 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.753112 4457 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 19:57:01.756735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.756712 4457 generic.go:356] "Generic (PLEG): container finished" podID="0616d8478d645504ab20b6a243f25dbb" containerID="589741ed9276485d832c4b782d1b14eb59a4c5ac86f2741859bda65c8afc1faf" exitCode=0 Nov 03 19:57:01.768783 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.768760 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 19:57:01.769096 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.769073 4457 generic.go:356] "Generic (PLEG): container finished" podID="af954863-9eb8-4bd6-a64b-918f5f4ff6b0" containerID="a000026611a4514c07cd1cc9067e28b492e536d864da11d11971173f90384cbb" exitCode=1 Nov 03 19:57:01.781454 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.781427 4457 generic.go:356] "Generic (PLEG): container finished" podID="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" containerID="8511e63cd28397e00df69b9a31c960cc21849edc7409cbe9d4f258ed2b709c9b" exitCode=0 Nov 03 19:57:01.781560 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.781455 4457 generic.go:356] "Generic (PLEG): container finished" podID="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" containerID="8400ed3b6e43b5ae4f65167de453975dc398ce9dbf85accb0c5392c9a6b34db8" exitCode=0 Nov 03 19:57:01.781560 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.781464 4457 generic.go:356] "Generic (PLEG): container finished" podID="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" containerID="9273dda4508acde0057b732234ef22156a38fbab9bd8c579c9d712cf72d55cf6" exitCode=0 Nov 03 19:57:01.781560 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.781473 4457 generic.go:356] "Generic (PLEG): container finished" podID="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" containerID="eedef9467086f2cdc20e645a9a90b0dc2d310c32789643a0586767fbf9d73f87" exitCode=0 Nov 03 19:57:01.781560 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.781481 4457 generic.go:356] "Generic (PLEG): container finished" podID="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" containerID="3ebfd424d879d07593d5bdcfb82f532fbee4a760f4180e3af4e6cc6517657154" exitCode=0 Nov 03 19:57:01.781560 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.781489 4457 generic.go:356] "Generic (PLEG): container finished" podID="c2c45b0c-9bd5-42bc-8a11-b2db15876d63" containerID="b83017975f7dfe5b6b6da6d7bde2ba5d209b522d5818aae9e397682e9294eae0" exitCode=0 Nov 03 19:57:01.786834 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.786817 4457 manager.go:324] Recovery completed Nov 03 19:57:01.801933 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.801880 4457 cpu_manager.go:226] "Starting" policy="none" Nov 03 19:57:01.801933 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.801895 4457 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 19:57:01.801933 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.801912 4457 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 19:57:01.802073 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.802030 4457 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 19:57:01.802073 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.802041 4457 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 19:57:01.802073 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.802058 4457 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 19:57:01.802073 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.802063 4457 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 19:57:01.802073 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.802068 4457 policy_none.go:50] "Start" Nov 03 19:57:01.802202 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.802086 4457 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 19:57:01.802202 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.802095 4457 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 19:57:01.802251 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.802217 4457 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 19:57:01.802251 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.802224 4457 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 19:57:01.802251 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.802229 4457 policy_none.go:44] "Start" Nov 03 19:57:01.806154 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.806130 4457 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 19:57:01.808721 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.808709 4457 manager.go:341] "Starting Device Plugin manager" Nov 03 19:57:01.809272 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:01.808732 4457 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 19:57:01.809306 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.809276 4457 server.go:87] "Starting device plugin registration server" Nov 03 19:57:01.809545 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.809532 4457 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 19:57:01.809575 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.809548 4457 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 19:57:01.809667 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.809648 4457 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 19:57:01.809858 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.809808 4457 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 19:57:01.809927 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.809866 4457 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 19:57:01.809927 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.809874 4457 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 19:57:01.810110 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.810010 4457 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T19:57:01.809855223Z","UUID":"4a7d3eda-c1d3-47e8-8216-f3aefe1f8e20","Handler":null,"Name":"","Endpoint":""} Nov 03 19:57:01.811141 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:01.811121 4457 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 19:57:01.811250 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.811240 4457 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 19:57:01.811313 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.811262 4457 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 19:57:01.849782 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.849744 4457 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal"] Nov 03 19:57:01.850264 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.850227 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" event={"ID":"0616d8478d645504ab20b6a243f25dbb","Type":"ContainerStarted","Data":"d74fc62c5607f6b02ce6ae75511f0601c705ab8aae3ebf8548121158022b8c10"} Nov 03 19:57:01.850314 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.850278 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" event={"ID":"0616d8478d645504ab20b6a243f25dbb","Type":"ContainerDied","Data":"589741ed9276485d832c4b782d1b14eb59a4c5ac86f2741859bda65c8afc1faf"} Nov 03 19:57:01.850314 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.850301 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" event={"ID":"0616d8478d645504ab20b6a243f25dbb","Type":"ContainerStarted","Data":"bd6773b2f462d346511693cad2335a5ba671730d1886dde1e05ec240083a3169"} Nov 03 19:57:01.850424 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.850340 4457 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:01.850424 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.850416 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" event={"ID":"ca7a77d780aa380ba43b79db4d32277a","Type":"ContainerStarted","Data":"7cd8ef78a2b0513babcc7e551998130cd6b72424cdcd67acaa1d02e70cef2fdc"} Nov 03 19:57:01.850481 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.850440 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" event={"ID":"ca7a77d780aa380ba43b79db4d32277a","Type":"ContainerStarted","Data":"35150edf568d96bc082d83b80c1db26cdb6a69c1259c86011542b4cc61f141a6"} Nov 03 19:57:01.850481 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.850449 4457 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:01.861615 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.861597 4457 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:57:01.861665 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:01.861632 4457 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:01.861797 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.861786 4457 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:57:01.861849 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:01.861829 4457 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:01.910014 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.909991 4457 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 19:57:01.911232 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.911217 4457 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 03 19:57:01.911303 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.911246 4457 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 03 19:57:01.911303 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.911256 4457 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-25.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 03 19:57:01.911303 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.911285 4457 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:01.921895 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.921874 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0616d8478d645504ab20b6a243f25dbb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"0616d8478d645504ab20b6a243f25dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:01.921974 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.921902 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0616d8478d645504ab20b6a243f25dbb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"0616d8478d645504ab20b6a243f25dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:01.921974 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.921920 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca7a77d780aa380ba43b79db4d32277a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"ca7a77d780aa380ba43b79db4d32277a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:01.925882 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.925863 4457 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:01.925950 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:01.925920 4457 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:02.023096 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.023061 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0616d8478d645504ab20b6a243f25dbb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"0616d8478d645504ab20b6a243f25dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:02.023238 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.023101 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0616d8478d645504ab20b6a243f25dbb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"0616d8478d645504ab20b6a243f25dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:02.023238 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.023130 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca7a77d780aa380ba43b79db4d32277a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"ca7a77d780aa380ba43b79db4d32277a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:02.023238 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.023169 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0616d8478d645504ab20b6a243f25dbb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"0616d8478d645504ab20b6a243f25dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:02.023238 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.023172 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ca7a77d780aa380ba43b79db4d32277a-config\") pod \"kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"ca7a77d780aa380ba43b79db4d32277a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:02.023238 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.023176 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0616d8478d645504ab20b6a243f25dbb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" (UID: \"0616d8478d645504ab20b6a243f25dbb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:02.696608 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.696568 4457 apiserver.go:52] "Watching apiserver" Nov 03 19:57:02.705316 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.705290 4457 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 19:57:02.705797 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.705757 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qxxzr","openshift-network-diagnostics/network-check-target-h4zrs","openshift-network-operator/iptables-alerter-zqftx","kube-system/konnectivity-agent-9ktt5","openshift-image-registry/node-ca-cwtc9","openshift-insights/insights-runtime-extractor-6tmzf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal","openshift-multus/multus-5p4wp","openshift-dns/dns-default-5zxlx","openshift-service-ca/service-ca-5fffc97bdf-zfvf2","kube-system/global-pull-secret-syncer-l5wn9","openshift-cluster-node-tuning-operator/tuned-q5hct","openshift-multus/multus-additional-cni-plugins-xx4kz","kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj","openshift-dns/node-resolver-ssj5l","openshift-multus/network-metrics-daemon-st6tw","openshift-ovn-kubernetes/ovnkube-node-sc9nt"] Nov 03 19:57:02.709726 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.709702 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:57:02.710500 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.710477 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6tnlf\"" Nov 03 19:57:02.710599 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.710544 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l9cjs\"" Nov 03 19:57:02.711260 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.711239 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4n975\"" Nov 03 19:57:02.711788 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.711762 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n8st4\"" Nov 03 19:57:02.711976 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.711961 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 19:57:02.712101 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.712080 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:57:02.712196 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.712182 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 19:57:02.712299 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.712285 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 19:57:02.712456 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.712443 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 19:57:02.712525 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.712465 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 19:57:02.712587 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.712548 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 19:57:02.712645 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.712090 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 19:57:02.712694 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.712666 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5vrk4\"" Nov 03 19:57:02.712748 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.712706 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 19:57:02.712845 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.712823 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 19:57:02.713046 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.713027 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 19:57:02.713144 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.713126 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-gc46m\"" Nov 03 19:57:02.713222 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.713206 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 19:57:02.713443 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.713428 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-72cwb\"" Nov 03 19:57:02.713511 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.713479 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 19:57:02.713665 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.713644 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-hc4hl\"" Nov 03 19:57:02.713835 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.713811 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 19:57:02.714715 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.714667 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 19:57:02.714715 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.714713 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 19:57:02.714864 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.714739 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 19:57:02.714864 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.714752 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 19:57:02.714864 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.714777 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 19:57:02.714864 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.714853 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 19:57:02.715075 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.714892 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 19:57:02.715129 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.715108 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:02.715176 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.715142 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 19:57:02.715256 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.715241 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lp2l4\"" Nov 03 19:57:02.715305 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.715292 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 19:57:02.715488 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.715474 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 19:57:02.715543 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.715523 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sztcx\"" Nov 03 19:57:02.715604 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.715576 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 19:57:02.715746 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.715732 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 19:57:02.716413 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.716398 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:02.718791 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.718772 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 19:57:02.718791 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.718789 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 19:57:02.718902 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.718795 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 19:57:02.718902 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.718800 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 19:57:02.718902 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.718846 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wsxsf\"" Nov 03 19:57:02.719046 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.719034 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 19:57:02.719078 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.719045 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 19:57:02.719108 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.719094 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 19:57:02.719273 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.719255 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 19:57:02.719273 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.719262 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 19:57:02.719404 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.719310 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 19:57:02.719404 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.719315 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 19:57:02.719404 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.719350 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-p44lh\"" Nov 03 19:57:02.719744 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.719730 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 19:57:02.720018 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.720006 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 19:57:02.720073 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.720056 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 19:57:02.720252 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.720232 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 19:57:02.720252 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.720243 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-f759x\"" Nov 03 19:57:02.720339 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.720251 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 19:57:02.720339 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.720252 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 19:57:02.720339 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.720309 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kfdzx\"" Nov 03 19:57:02.720339 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.720309 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rpk57\"" Nov 03 19:57:02.720506 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.720365 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4mwrd\"" Nov 03 19:57:02.720506 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.720377 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 19:57:02.726534 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.726518 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-multus-certs\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.726534 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.726541 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-dbus\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:57:02.726534 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.726556 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-cnibin\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.726792 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.726575 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kwj4\" (UniqueName: \"kubernetes.io/projected/2590e213-dc39-406a-8094-b4e01162e221-kube-api-access-8kwj4\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.726792 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.726595 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-k8s-cni-cncf-io\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.726792 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.726677 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-run-netns\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.726792 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.726707 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-etc-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.726792 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.726756 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-socket-dir-parent\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.727037 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.726800 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-kubelet\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.727037 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.726828 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-slash\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.727037 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.726881 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-env-overrides\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.727037 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.726908 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cni-binary-copy\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.727217 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727118 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-env-overrides\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.727217 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727158 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.727217 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727208 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-system-cni-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.727315 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727248 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cni-binary-copy\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.727315 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727249 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:57:02.727377 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727364 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/642aaac2-4dc2-4800-bc55-c70730884400-konnectivity-ca\") pod \"konnectivity-agent-9ktt5\" (UID: \"642aaac2-4dc2-4800-bc55-c70730884400\") " pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:57:02.727451 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727413 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-cni-netd\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.727559 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727524 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:57:02.727559 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727550 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-original-pull-secret\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:57:02.727676 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727579 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2590e213-dc39-406a-8094-b4e01162e221-cni-binary-copy\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.727676 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727603 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-cni-multus\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.727676 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727650 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/642aaac2-4dc2-4800-bc55-c70730884400-konnectivity-ca\") pod \"konnectivity-agent-9ktt5\" (UID: \"642aaac2-4dc2-4800-bc55-c70730884400\") " pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:57:02.727816 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727677 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2590e213-dc39-406a-8094-b4e01162e221-multus-daemon-config\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.727816 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727715 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz2q2\" (UniqueName: \"kubernetes.io/projected/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-kube-api-access-wz2q2\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.727816 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727755 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn92t\" (UniqueName: \"kubernetes.io/projected/33806d61-5ac2-49ba-9fc9-c89046f2ff03-kube-api-access-tn92t\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:57:02.727816 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727765 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2590e213-dc39-406a-8094-b4e01162e221-cni-binary-copy\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.727816 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727781 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-cni-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.728005 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727807 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-netns\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.728005 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727855 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-systemd\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.728005 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727869 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2590e213-dc39-406a-8094-b4e01162e221-multus-daemon-config\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.728005 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727900 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.728005 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727948 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cnibin\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.728005 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.727985 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.728286 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728033 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b33b71b3-287e-4516-9d0a-726f302a8dff-host-slash\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:57:02.728286 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728053 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffa0c959-6bd1-4feb-b713-3545a6896409-host\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:57:02.728286 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728070 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-etc-kubernetes\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.728286 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728084 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.728286 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728102 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b33b71b3-287e-4516-9d0a-726f302a8dff-iptables-alerter-script\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:57:02.728286 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728116 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-cni-bin\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.728286 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728146 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/642aaac2-4dc2-4800-bc55-c70730884400-agent-certs\") pod \"konnectivity-agent-9ktt5\" (UID: \"642aaac2-4dc2-4800-bc55-c70730884400\") " pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:57:02.728286 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728184 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.728286 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728206 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-var-lib-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.728286 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728275 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovnkube-script-lib\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728292 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jw95\" (UniqueName: \"kubernetes.io/projected/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-kube-api-access-7jw95\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728315 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-cni-bin\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728356 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovnkube-config\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728372 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b33b71b3-287e-4516-9d0a-726f302a8dff-iptables-alerter-script\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728405 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728432 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/642aaac2-4dc2-4800-bc55-c70730884400-agent-certs\") pod \"konnectivity-agent-9ktt5\" (UID: \"642aaac2-4dc2-4800-bc55-c70730884400\") " pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728446 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-ovn\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728488 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovnkube-script-lib\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728490 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qspj\" (UniqueName: \"kubernetes.io/projected/b33b71b3-287e-4516-9d0a-726f302a8dff-kube-api-access-8qspj\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728556 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hggp5\" (UniqueName: \"kubernetes.io/projected/ffa0c959-6bd1-4feb-b713-3545a6896409-kube-api-access-hggp5\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728564 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728582 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-kubelet-config\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728611 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728643 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-node-log\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728654 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovnkube-config\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728700 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-system-cni-dir\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728724 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovn-node-metrics-cert\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.728759 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728755 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-hostroot\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.729301 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728776 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75qtn\" (UniqueName: \"kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn\") pod \"network-check-target-h4zrs\" (UID: \"c98e82ab-c3b3-4d18-b381-083fdb90b448\") " pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:57:02.729301 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728792 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-log-socket\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.729301 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728812 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-kubelet\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.729301 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728830 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-systemd-units\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.729301 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728847 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffa0c959-6bd1-4feb-b713-3545a6896409-serviceca\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:57:02.729301 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728883 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-conf-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.729301 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.728980 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-ovn-node-metrics-cert\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.729301 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.729080 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ffa0c959-6bd1-4feb-b713-3545a6896409-serviceca\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:57:02.736842 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.736824 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kwj4\" (UniqueName: \"kubernetes.io/projected/2590e213-dc39-406a-8094-b4e01162e221-kube-api-access-8kwj4\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.737581 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.737562 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn92t\" (UniqueName: \"kubernetes.io/projected/33806d61-5ac2-49ba-9fc9-c89046f2ff03-kube-api-access-tn92t\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:57:02.737850 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.737833 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz2q2\" (UniqueName: \"kubernetes.io/projected/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-kube-api-access-wz2q2\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.738533 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.738511 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qspj\" (UniqueName: \"kubernetes.io/projected/b33b71b3-287e-4516-9d0a-726f302a8dff-kube-api-access-8qspj\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:57:02.738613 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.738511 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hggp5\" (UniqueName: \"kubernetes.io/projected/ffa0c959-6bd1-4feb-b713-3545a6896409-kube-api-access-hggp5\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:57:02.738764 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.738746 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jw95\" (UniqueName: \"kubernetes.io/projected/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-kube-api-access-7jw95\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.739087 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.739066 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75qtn\" (UniqueName: \"kubernetes.io/projected/c98e82ab-c3b3-4d18-b381-083fdb90b448-kube-api-access-75qtn\") pod \"network-check-target-h4zrs\" (UID: \"c98e82ab-c3b3-4d18-b381-083fdb90b448\") " pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:57:02.785930 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.785902 4457 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:02.786051 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.785969 4457 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:02.797882 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.797858 4457 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:57:02.798032 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:02.797903 4457 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:02.798325 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.798311 4457 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 19:57:02.798414 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:02.798348 4457 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-25.us-east-2.compute.internal" Nov 03 19:57:02.809848 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.809828 4457 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 19:57:02.830080 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830034 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cnibin\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.830227 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830104 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-registration-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.830227 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830149 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysconfig\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.830227 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830190 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b33b71b3-287e-4516-9d0a-726f302a8dff-host-slash\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:57:02.830409 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830245 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffa0c959-6bd1-4feb-b713-3545a6896409-host\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:57:02.830409 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830276 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-etc-kubernetes\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.830409 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830314 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.830409 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830350 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-kubernetes\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.830409 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830403 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-lib-modules\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.830637 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830448 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-cni-bin\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.830637 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830483 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-var-lib-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.830637 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830523 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-device-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.830637 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830591 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/224062e4-7b08-4d73-ba03-1789a9f591be-data-volume\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:02.830637 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830615 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ffa0c959-6bd1-4feb-b713-3545a6896409-host\") pod \"node-ca-cwtc9\" (UID: \"ffa0c959-6bd1-4feb-b713-3545a6896409\") " pod="openshift-image-registry/node-ca-cwtc9" Nov 03 19:57:02.830637 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830633 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-cni-bin\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.830876 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830669 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-config-volume\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:02.830876 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830687 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-run-ovn-kubernetes\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.830876 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830697 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-var-lib-kubelet\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.830876 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830768 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lplg9\" (UniqueName: \"kubernetes.io/projected/308feccf-7178-48c7-b370-3c78f96e8919-kube-api-access-lplg9\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.830876 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830811 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-cnibin\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.830876 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830860 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-cni-bin\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.831133 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830893 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-ovn\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.831133 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830916 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/224062e4-7b08-4d73-ba03-1789a9f591be-data-volume\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:02.831133 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830927 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/224062e4-7b08-4d73-ba03-1789a9f591be-crio-socket\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:02.831133 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.830966 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czx85\" (UniqueName: \"kubernetes.io/projected/97d28296-595f-4a77-926d-f6c30841085f-kube-api-access-czx85\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:57:02.831133 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831013 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-kubelet-config\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:57:02.831133 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831044 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.831133 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831076 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-node-log\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.831133 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831119 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-system-cni-dir\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.831495 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831155 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvxvg\" (UniqueName: \"kubernetes.io/projected/224062e4-7b08-4d73-ba03-1789a9f591be-kube-api-access-nvxvg\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:02.831495 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831186 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqljf\" (UniqueName: \"kubernetes.io/projected/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-kube-api-access-dqljf\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:02.831495 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831220 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-hostroot\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.831495 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831253 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-config-volume\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:02.831495 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831246 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-log-socket\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.831495 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831014 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-cni-bin\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.831495 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831377 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-etc-kubernetes\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.831495 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831374 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b33b71b3-287e-4516-9d0a-726f302a8dff-host-slash\") pod \"iptables-alerter-zqftx\" (UID: \"b33b71b3-287e-4516-9d0a-726f302a8dff\") " pod="openshift-network-operator/iptables-alerter-zqftx" Nov 03 19:57:02.831495 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831440 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-node-log\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.831841 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831524 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-kubelet-config\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:57:02.831841 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831570 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.831841 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831611 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-hostroot\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.831841 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831629 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-ovn\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.831841 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831644 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-socket-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.831841 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831684 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-system-cni-dir\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.831841 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831719 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-var-lib-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.832117 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831886 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-log-socket\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.832117 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831960 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:02.832117 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.831993 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f215e51-f5cf-4103-9904-693d5721e647-tmp-dir\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:57:02.832117 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832055 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-kubelet\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.832117 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832087 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-systemd-units\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.832345 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832119 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac15ca75-0dd3-4f05-b21d-1ae2280d0415-cert\") pod \"ingress-canary-qxxzr\" (UID: \"ac15ca75-0dd3-4f05-b21d-1ae2280d0415\") " pod="openshift-ingress-canary/ingress-canary-qxxzr" Nov 03 19:57:02.832345 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832150 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f215e51-f5cf-4103-9904-693d5721e647-hosts-file\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:57:02.832345 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832184 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f215e51-f5cf-4103-9904-693d5721e647-tmp-dir\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:57:02.832345 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832263 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:02.832345 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832314 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-conf-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.832566 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832349 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-multus-certs\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.832566 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832354 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/224062e4-7b08-4d73-ba03-1789a9f591be-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:02.832566 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832420 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm4bm\" (UniqueName: \"kubernetes.io/projected/ac15ca75-0dd3-4f05-b21d-1ae2280d0415-kube-api-access-qm4bm\") pod \"ingress-canary-qxxzr\" (UID: \"ac15ca75-0dd3-4f05-b21d-1ae2280d0415\") " pod="openshift-ingress-canary/ingress-canary-qxxzr" Nov 03 19:57:02.832566 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832458 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:02.832566 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832492 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-tmp-dir\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:02.832566 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832493 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-multus-certs\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.832566 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832459 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-systemd-units\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.832566 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832422 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-kubelet\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.832566 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832524 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-modprobe-d\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.832925 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832599 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-tmp-dir\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:02.832925 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832647 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-conf-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.832925 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832787 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-sys\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.832925 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832821 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-dbus\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:57:02.832925 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832859 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-cnibin\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.832925 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832868 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac15ca75-0dd3-4f05-b21d-1ae2280d0415-cert\") pod \"ingress-canary-qxxzr\" (UID: \"ac15ca75-0dd3-4f05-b21d-1ae2280d0415\") " pod="openshift-ingress-canary/ingress-canary-qxxzr" Nov 03 19:57:02.833167 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832937 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e6338d09-ef6a-4cf5-8a06-bd466bc5076d-dbus\") pod \"global-pull-secret-syncer-l5wn9\" (UID: \"e6338d09-ef6a-4cf5-8a06-bd466bc5076d\") " pod="kube-system/global-pull-secret-syncer-l5wn9" Nov 03 19:57:02.833167 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832928 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbht\" (UniqueName: \"kubernetes.io/projected/7f215e51-f5cf-4103-9904-693d5721e647-kube-api-access-mgbht\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:57:02.833167 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.832988 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97d28296-595f-4a77-926d-f6c30841085f-signing-cabundle\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:57:02.833167 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833002 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-cnibin\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.833167 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833137 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysctl-d\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.833378 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833209 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-k8s-cni-cncf-io\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.833378 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833236 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97d28296-595f-4a77-926d-f6c30841085f-signing-cabundle\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:57:02.833378 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833244 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-run-netns\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.833378 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833295 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-run-netns\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.833378 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833306 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-etc-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.833378 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833309 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-k8s-cni-cncf-io\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.833378 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833370 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-etc-openvswitch\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.833687 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833425 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.833687 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833469 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.833687 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833505 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-sys-fs\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.833687 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833564 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-socket-dir-parent\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.833687 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833592 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-kubelet\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.833687 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833645 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-slash\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.833687 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833671 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.834002 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833699 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-system-cni-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.834002 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833744 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-kubelet\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.834002 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833823 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-socket-dir-parent\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.834002 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833867 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-slash\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.834002 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833925 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2c45b0c-9bd5-42bc-8a11-b2db15876d63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xx4kz\" (UID: \"c2c45b0c-9bd5-42bc-8a11-b2db15876d63\") " pod="openshift-multus/multus-additional-cni-plugins-xx4kz" Nov 03 19:57:02.834002 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.833990 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-system-cni-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.834243 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.834043 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-cni-netd\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.834243 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.834094 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:57:02.834243 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.834148 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-cni-netd\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.834243 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.834229 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp952\" (UniqueName: \"kubernetes.io/projected/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-kube-api-access-fp952\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.834430 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.834284 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97d28296-595f-4a77-926d-f6c30841085f-signing-key\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.834593 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysctl-conf\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.834736 4457 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.834843 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97d28296-595f-4a77-926d-f6c30841085f-signing-key\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.834905 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-cni-multus\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.834972 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-systemd\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835009 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-host\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835054 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-run\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835096 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/308feccf-7178-48c7-b370-3c78f96e8919-etc-tuned\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835126 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/308feccf-7178-48c7-b370-3c78f96e8919-tmp\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835177 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-cni-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835210 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-netns\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835245 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-systemd\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835336 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835425 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835463 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/af954863-9eb8-4bd6-a64b-918f5f4ff6b0-run-systemd\") pod \"ovnkube-node-sc9nt\" (UID: \"af954863-9eb8-4bd6-a64b-918f5f4ff6b0\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835507 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-var-lib-cni-multus\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835557 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-multus-cni-dir\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835555 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/308feccf-7178-48c7-b370-3c78f96e8919-tmp\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835621 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2590e213-dc39-406a-8094-b4e01162e221-host-run-netns\") pod \"multus-5p4wp\" (UID: \"2590e213-dc39-406a-8094-b4e01162e221\") " pod="openshift-multus/multus-5p4wp" Nov 03 19:57:02.835735 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.835634 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/308feccf-7178-48c7-b370-3c78f96e8919-etc-tuned\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.837252 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.837234 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33806d61-5ac2-49ba-9fc9-c89046f2ff03-metrics-certs\") pod \"network-metrics-daemon-st6tw\" (UID: \"33806d61-5ac2-49ba-9fc9-c89046f2ff03\") " pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:57:02.841806 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.841769 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqljf\" (UniqueName: \"kubernetes.io/projected/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-kube-api-access-dqljf\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:02.841806 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.841793 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvxvg\" (UniqueName: \"kubernetes.io/projected/224062e4-7b08-4d73-ba03-1789a9f591be-kube-api-access-nvxvg\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:02.842208 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.842184 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czx85\" (UniqueName: \"kubernetes.io/projected/97d28296-595f-4a77-926d-f6c30841085f-kube-api-access-czx85\") pod \"service-ca-5fffc97bdf-zfvf2\" (UID: \"97d28296-595f-4a77-926d-f6c30841085f\") " pod="openshift-service-ca/service-ca-5fffc97bdf-zfvf2" Nov 03 19:57:02.842345 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.842327 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lplg9\" (UniqueName: \"kubernetes.io/projected/308feccf-7178-48c7-b370-3c78f96e8919-kube-api-access-lplg9\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.842424 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.842362 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm4bm\" (UniqueName: \"kubernetes.io/projected/ac15ca75-0dd3-4f05-b21d-1ae2280d0415-kube-api-access-qm4bm\") pod \"ingress-canary-qxxzr\" (UID: \"ac15ca75-0dd3-4f05-b21d-1ae2280d0415\") " pod="openshift-ingress-canary/ingress-canary-qxxzr" Nov 03 19:57:02.842853 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.842834 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbht\" (UniqueName: \"kubernetes.io/projected/7f215e51-f5cf-4103-9904-693d5721e647-kube-api-access-mgbht\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:57:02.844603 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.844585 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp952\" (UniqueName: \"kubernetes.io/projected/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-kube-api-access-fp952\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.936376 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936270 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-modprobe-d\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936376 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936315 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-sys\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936376 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936337 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysctl-d\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936376 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936361 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.936376 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936383 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936435 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-sys-fs\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936448 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-sys\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936466 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysctl-conf\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936473 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-modprobe-d\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936489 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-systemd\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936495 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-sys-fs\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936482 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysctl-d\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936508 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-host\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936452 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936508 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-etc-selinux\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936550 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-run\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936559 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-systemd\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936565 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysctl-conf\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936575 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-registration-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936564 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-host\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936588 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-run\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936592 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysconfig\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936644 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-sysconfig\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936645 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-registration-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936663 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-kubernetes\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936689 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-lib-modules\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.936698 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936706 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-device-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936729 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-var-lib-kubelet\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936745 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/224062e4-7b08-4d73-ba03-1789a9f591be-crio-socket\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936746 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-etc-kubernetes\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936764 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-lib-modules\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936766 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-socket-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936795 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f215e51-f5cf-4103-9904-693d5721e647-hosts-file\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936797 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/308feccf-7178-48c7-b370-3c78f96e8919-var-lib-kubelet\") pod \"tuned-q5hct\" (UID: \"308feccf-7178-48c7-b370-3c78f96e8919\") " pod="openshift-cluster-node-tuning-operator/tuned-q5hct" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936799 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/224062e4-7b08-4d73-ba03-1789a9f591be-crio-socket\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936828 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-device-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936833 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7f215e51-f5cf-4103-9904-693d5721e647-hosts-file\") pod \"node-resolver-ssj5l\" (UID: \"7f215e51-f5cf-4103-9904-693d5721e647\") " pod="openshift-dns/node-resolver-ssj5l" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936867 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc075c5d-3e03-4e5c-be0d-09efa41fcdb9-socket-dir\") pod \"aws-ebs-csi-driver-node-s4wjj\" (UID: \"cc075c5d-3e03-4e5c-be0d-09efa41fcdb9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s4wjj" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936866 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:02.936900 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:02.936944 4457 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:02.936979 4457 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:02.937026 4457 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls podName:731a10e2-8c1d-48c4-a453-1d1d589f6ae2 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:03.437004284 +0000 UTC m=+1.812601029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls") pod "dns-default-5zxlx" (UID: "731a10e2-8c1d-48c4-a453-1d1d589f6ae2") : secret "dns-default-metrics-tls" not found Nov 03 19:57:02.937352 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:02.937067 4457 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls podName:224062e4-7b08-4d73-ba03-1789a9f591be nodeName:}" failed. No retries permitted until 2026-11-03 19:57:03.437055965 +0000 UTC m=+1.812652703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tmzf" (UID: "224062e4-7b08-4d73-ba03-1789a9f591be") : secret "insights-runtime-extractor-tls" not found Nov 03 19:57:03.010807 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:03.010775 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-st6tw" Nov 03 19:57:03.278023 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:03.277938 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:57:03.316951 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:03.316924 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-st6tw"] Nov 03 19:57:03.319962 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:03.319921 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33806d61_5ac2_49ba_9fc9_c89046f2ff03.slice/crio-4b4af3881878061eed63bbed4fb085db17f4d0a33818b898d97bdf0245d16368 WatchSource:0}: Error finding container 4b4af3881878061eed63bbed4fb085db17f4d0a33818b898d97bdf0245d16368: Status 404 returned error can't find the container with id 4b4af3881878061eed63bbed4fb085db17f4d0a33818b898d97bdf0245d16368 Nov 03 19:57:03.322199 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:03.322179 4457 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 19:57:03.440367 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:03.440328 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:03.440546 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:03.440423 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:03.440546 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:03.440492 4457 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 19:57:03.440546 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:03.440538 4457 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:57:03.440664 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:03.440553 4457 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls podName:731a10e2-8c1d-48c4-a453-1d1d589f6ae2 nodeName:}" failed. No retries permitted until 2026-11-03 19:57:04.440539452 +0000 UTC m=+2.816136177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls") pod "dns-default-5zxlx" (UID: "731a10e2-8c1d-48c4-a453-1d1d589f6ae2") : secret "dns-default-metrics-tls" not found Nov 03 19:57:03.440664 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:03.440585 4457 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls podName:224062e4-7b08-4d73-ba03-1789a9f591be nodeName:}" failed. No retries permitted until 2026-11-03 19:57:04.440573762 +0000 UTC m=+2.816170487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tmzf" (UID: "224062e4-7b08-4d73-ba03-1789a9f591be") : secret "insights-runtime-extractor-tls" not found Nov 03 19:57:03.789463 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:03.789430 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-st6tw" event={"ID":"33806d61-5ac2-49ba-9fc9-c89046f2ff03","Type":"ContainerStarted","Data":"4b4af3881878061eed63bbed4fb085db17f4d0a33818b898d97bdf0245d16368"} Nov 03 19:57:04.448101 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:04.448074 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:04.448344 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:04.448121 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:04.448344 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:04.448226 4457 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:57:04.448344 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:04.448300 4457 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls podName:224062e4-7b08-4d73-ba03-1789a9f591be nodeName:}" failed. No retries permitted until 2026-11-03 19:57:06.448269467 +0000 UTC m=+4.823866196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tmzf" (UID: "224062e4-7b08-4d73-ba03-1789a9f591be") : secret "insights-runtime-extractor-tls" not found Nov 03 19:57:04.450468 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:04.450440 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/731a10e2-8c1d-48c4-a453-1d1d589f6ae2-metrics-tls\") pod \"dns-default-5zxlx\" (UID: \"731a10e2-8c1d-48c4-a453-1d1d589f6ae2\") " pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:04.515951 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:04.515922 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:04.790890 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:04.790816 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5zxlx"] Nov 03 19:57:04.794181 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:04.794142 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-st6tw" event={"ID":"33806d61-5ac2-49ba-9fc9-c89046f2ff03","Type":"ContainerStarted","Data":"1ff156328577afa458456376dab5a906ce25dced63b72640c54337cede7a28de"} Nov 03 19:57:04.794285 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:04.794193 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-st6tw" event={"ID":"33806d61-5ac2-49ba-9fc9-c89046f2ff03","Type":"ContainerStarted","Data":"9ddb40d6ee45daec391fc3a0e7e084683c770b3ffde3bd690183b19a6b47b907"} Nov 03 19:57:04.794409 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:04.794370 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731a10e2_8c1d_48c4_a453_1d1d589f6ae2.slice/crio-acdc44e2521c148f71c70d9ca04febd77c032a21f7c6c34bcf6ec1e7e558754f WatchSource:0}: Error finding container acdc44e2521c148f71c70d9ca04febd77c032a21f7c6c34bcf6ec1e7e558754f: Status 404 returned error can't find the container with id acdc44e2521c148f71c70d9ca04febd77c032a21f7c6c34bcf6ec1e7e558754f Nov 03 19:57:05.799550 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:05.799518 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5zxlx" event={"ID":"731a10e2-8c1d-48c4-a453-1d1d589f6ae2","Type":"ContainerStarted","Data":"acdc44e2521c148f71c70d9ca04febd77c032a21f7c6c34bcf6ec1e7e558754f"} Nov 03 19:57:06.466500 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:06.466466 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:06.466638 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:06.466613 4457 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 19:57:06.466695 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:06.466681 4457 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls podName:224062e4-7b08-4d73-ba03-1789a9f591be nodeName:}" failed. No retries permitted until 2026-11-03 19:57:10.466665174 +0000 UTC m=+8.842261912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6tmzf" (UID: "224062e4-7b08-4d73-ba03-1789a9f591be") : secret "insights-runtime-extractor-tls" not found Nov 03 19:57:06.803962 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:06.803927 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5zxlx" event={"ID":"731a10e2-8c1d-48c4-a453-1d1d589f6ae2","Type":"ContainerStarted","Data":"7006c1fdbf96bbd9c1a8071bd307e51bd5d3b2f9f94f42e378c92c5fd5dd4080"} Nov 03 19:57:06.804361 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:06.803983 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5zxlx" event={"ID":"731a10e2-8c1d-48c4-a453-1d1d589f6ae2","Type":"ContainerStarted","Data":"086a22683476e178fba06233763b07d1feecf0f918fdaeffa7ff521699acfcd3"} Nov 03 19:57:06.804361 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:06.804087 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:07.943799 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:07.943771 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9ktt5" Nov 03 19:57:10.393584 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:10.393554 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:10.491129 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:10.491082 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:10.493457 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:10.493429 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/224062e4-7b08-4d73-ba03-1789a9f591be-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6tmzf\" (UID: \"224062e4-7b08-4d73-ba03-1789a9f591be\") " pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:10.517396 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:10.517367 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6tmzf" Nov 03 19:57:10.786365 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:10.786298 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6tmzf"] Nov 03 19:57:10.789362 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:10.789337 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224062e4_7b08_4d73_ba03_1789a9f591be.slice/crio-a0af90ef8e886f367ab681d7208aa66c700c636cc48aba3177f1dc03f87d854b WatchSource:0}: Error finding container a0af90ef8e886f367ab681d7208aa66c700c636cc48aba3177f1dc03f87d854b: Status 404 returned error can't find the container with id a0af90ef8e886f367ab681d7208aa66c700c636cc48aba3177f1dc03f87d854b Nov 03 19:57:10.816158 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:10.816131 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6tmzf" event={"ID":"224062e4-7b08-4d73-ba03-1789a9f591be","Type":"ContainerStarted","Data":"a0af90ef8e886f367ab681d7208aa66c700c636cc48aba3177f1dc03f87d854b"} Nov 03 19:57:11.822008 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:11.821966 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6tmzf" event={"ID":"224062e4-7b08-4d73-ba03-1789a9f591be","Type":"ContainerStarted","Data":"c1d42093593f98ea82dea48c62124c0c985acd0dbac2a942eafeebb09ac7367d"} Nov 03 19:57:11.822482 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:11.822051 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6tmzf" event={"ID":"224062e4-7b08-4d73-ba03-1789a9f591be","Type":"ContainerStarted","Data":"58c303a882037d9ec7bf48749f5cd5b5e7c5e920c2c1e3aa091b0ec89374f34a"} Nov 03 19:57:12.235647 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:12.235573 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:12.235800 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:12.235788 4457 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 19:57:13.818511 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.818488 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-nbjzs"] Nov 03 19:57:13.822261 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.822243 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rwwdh"] Nov 03 19:57:13.822424 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.822377 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nbjzs" Nov 03 19:57:13.824105 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.824088 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rwwdh" Nov 03 19:57:13.833084 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.833056 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bn5rk"] Nov 03 19:57:13.833847 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.833829 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 03 19:57:13.834094 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.833964 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 03 19:57:13.834335 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.834313 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Nov 03 19:57:13.834494 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.834418 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-bw6nv\"" Nov 03 19:57:13.834576 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.834492 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mhzkm\"" Nov 03 19:57:13.835369 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.835348 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bn5rk" Nov 03 19:57:13.836021 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.835979 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rwwdh"] Nov 03 19:57:13.838616 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.838596 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nbjzs"] Nov 03 19:57:13.849524 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.849504 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 19:57:13.850130 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.850102 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bn5rk"] Nov 03 19:57:13.850243 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.850224 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 19:57:13.900042 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.900018 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67895ffd5b-vwr6d"] Nov 03 19:57:13.902061 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.902043 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:13.914780 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.914759 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dac45c98-1ae5-4137-bc74-943c81ac477b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rwwdh\" (UID: \"dac45c98-1ae5-4137-bc74-943c81ac477b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rwwdh" Nov 03 19:57:13.914878 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.914785 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glgxb\" (UniqueName: \"kubernetes.io/projected/e331a233-4c29-4910-8bca-06f847a906d6-kube-api-access-glgxb\") pod \"migrator-66df5d496b-bn5rk\" (UID: \"e331a233-4c29-4910-8bca-06f847a906d6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bn5rk" Nov 03 19:57:13.914878 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.914810 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmk7w\" (UniqueName: \"kubernetes.io/projected/865485da-2a04-435a-8b33-18beb925bf8e-kube-api-access-cmk7w\") pod \"downloads-6dd789c8b7-nbjzs\" (UID: \"865485da-2a04-435a-8b33-18beb925bf8e\") " pod="openshift-console/downloads-6dd789c8b7-nbjzs" Nov 03 19:57:13.914961 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.914937 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/865485da-2a04-435a-8b33-18beb925bf8e-tmp\") pod \"downloads-6dd789c8b7-nbjzs\" (UID: \"865485da-2a04-435a-8b33-18beb925bf8e\") " pod="openshift-console/downloads-6dd789c8b7-nbjzs" Nov 03 19:57:13.918514 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.918495 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67895ffd5b-vwr6d"] Nov 03 19:57:13.923824 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.923804 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 19:57:13.933563 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.933498 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 19:57:13.933861 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.933814 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 19:57:13.933861 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.933816 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8rzcq\"" Nov 03 19:57:13.933955 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.933884 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6pnl5\"" Nov 03 19:57:13.933985 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:13.933967 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 19:57:14.015866 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.015832 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/865485da-2a04-435a-8b33-18beb925bf8e-tmp\") pod \"downloads-6dd789c8b7-nbjzs\" (UID: \"865485da-2a04-435a-8b33-18beb925bf8e\") " pod="openshift-console/downloads-6dd789c8b7-nbjzs" Nov 03 19:57:14.015866 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.015868 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-image-registry-private-configuration\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.016066 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.015885 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-installation-pull-secrets\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.016066 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.015906 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dac45c98-1ae5-4137-bc74-943c81ac477b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rwwdh\" (UID: \"dac45c98-1ae5-4137-bc74-943c81ac477b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rwwdh" Nov 03 19:57:14.016066 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.015924 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glgxb\" (UniqueName: \"kubernetes.io/projected/e331a233-4c29-4910-8bca-06f847a906d6-kube-api-access-glgxb\") pod \"migrator-66df5d496b-bn5rk\" (UID: \"e331a233-4c29-4910-8bca-06f847a906d6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bn5rk" Nov 03 19:57:14.016066 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.016005 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-trusted-ca\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.016066 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.016036 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-bound-sa-token\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.016066 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.016063 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2htf\" (UniqueName: \"kubernetes.io/projected/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-kube-api-access-t2htf\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.016301 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.016093 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmk7w\" (UniqueName: \"kubernetes.io/projected/865485da-2a04-435a-8b33-18beb925bf8e-kube-api-access-cmk7w\") pod \"downloads-6dd789c8b7-nbjzs\" (UID: \"865485da-2a04-435a-8b33-18beb925bf8e\") " pod="openshift-console/downloads-6dd789c8b7-nbjzs" Nov 03 19:57:14.016301 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.016289 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/865485da-2a04-435a-8b33-18beb925bf8e-tmp\") pod \"downloads-6dd789c8b7-nbjzs\" (UID: \"865485da-2a04-435a-8b33-18beb925bf8e\") " pod="openshift-console/downloads-6dd789c8b7-nbjzs" Nov 03 19:57:14.016371 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.016350 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-ca-trust-extracted\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.016427 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.016376 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-registry-tls\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.016427 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.016406 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-registry-certificates\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.018337 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.018318 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/dac45c98-1ae5-4137-bc74-943c81ac477b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-rwwdh\" (UID: \"dac45c98-1ae5-4137-bc74-943c81ac477b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rwwdh" Nov 03 19:57:14.030196 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.030174 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glgxb\" (UniqueName: \"kubernetes.io/projected/e331a233-4c29-4910-8bca-06f847a906d6-kube-api-access-glgxb\") pod \"migrator-66df5d496b-bn5rk\" (UID: \"e331a233-4c29-4910-8bca-06f847a906d6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bn5rk" Nov 03 19:57:14.030353 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.030335 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmk7w\" (UniqueName: \"kubernetes.io/projected/865485da-2a04-435a-8b33-18beb925bf8e-kube-api-access-cmk7w\") pod \"downloads-6dd789c8b7-nbjzs\" (UID: \"865485da-2a04-435a-8b33-18beb925bf8e\") " pod="openshift-console/downloads-6dd789c8b7-nbjzs" Nov 03 19:57:14.117681 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.117595 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-registry-certificates\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.117681 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.117639 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-image-registry-private-configuration\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.117681 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.117656 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-installation-pull-secrets\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.117681 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.117680 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-trusted-ca\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.117995 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.117695 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-bound-sa-token\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.117995 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.117717 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2htf\" (UniqueName: \"kubernetes.io/projected/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-kube-api-access-t2htf\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.117995 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.117769 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-ca-trust-extracted\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.117995 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.117796 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-registry-tls\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.118320 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.118293 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-ca-trust-extracted\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.118701 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.118678 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-registry-certificates\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.118861 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.118803 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-trusted-ca\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.120507 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.120484 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-installation-pull-secrets\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.120655 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.120637 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-image-registry-private-configuration\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.120715 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.120696 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-registry-tls\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.131479 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.131459 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-bound-sa-token\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.131614 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.131598 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2htf\" (UniqueName: \"kubernetes.io/projected/655da7d5-abe0-4c81-98a0-c4e2bf21b0a2-kube-api-access-t2htf\") pod \"image-registry-67895ffd5b-vwr6d\" (UID: \"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2\") " pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.156924 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.156900 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nbjzs" Nov 03 19:57:14.176273 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.176251 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rwwdh" Nov 03 19:57:14.178490 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.178469 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bn5rk" Nov 03 19:57:14.213771 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.213552 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.441014 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.440937 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nbjzs"] Nov 03 19:57:14.444831 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:14.444795 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865485da_2a04_435a_8b33_18beb925bf8e.slice/crio-28176f64d69b50e7bcf1bc6395afa7958a236efcc1d0634aefba5e2a97f04fd8 WatchSource:0}: Error finding container 28176f64d69b50e7bcf1bc6395afa7958a236efcc1d0634aefba5e2a97f04fd8: Status 404 returned error can't find the container with id 28176f64d69b50e7bcf1bc6395afa7958a236efcc1d0634aefba5e2a97f04fd8 Nov 03 19:57:14.470418 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.470369 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-bn5rk"] Nov 03 19:57:14.473509 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:14.473483 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode331a233_4c29_4910_8bca_06f847a906d6.slice/crio-cb1a49696ae633bad18ea1aa26993acc21f84938a10d224abb35d31bd6a90cb4 WatchSource:0}: Error finding container cb1a49696ae633bad18ea1aa26993acc21f84938a10d224abb35d31bd6a90cb4: Status 404 returned error can't find the container with id cb1a49696ae633bad18ea1aa26993acc21f84938a10d224abb35d31bd6a90cb4 Nov 03 19:57:14.475610 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.475590 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rwwdh"] Nov 03 19:57:14.478924 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:14.478901 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac45c98_1ae5_4137_bc74_943c81ac477b.slice/crio-eebfaf07f94d78ef5d26e91397e4e1aaf8cd9c3bb0e4a4e5421b3e7442046c8c WatchSource:0}: Error finding container eebfaf07f94d78ef5d26e91397e4e1aaf8cd9c3bb0e4a4e5421b3e7442046c8c: Status 404 returned error can't find the container with id eebfaf07f94d78ef5d26e91397e4e1aaf8cd9c3bb0e4a4e5421b3e7442046c8c Nov 03 19:57:14.509084 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.509056 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67895ffd5b-vwr6d"] Nov 03 19:57:14.512070 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:14.512037 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655da7d5_abe0_4c81_98a0_c4e2bf21b0a2.slice/crio-787cd666e097c65da9bae7838050dba6a2022c32d9a6794631b1979978bfb890 WatchSource:0}: Error finding container 787cd666e097c65da9bae7838050dba6a2022c32d9a6794631b1979978bfb890: Status 404 returned error can't find the container with id 787cd666e097c65da9bae7838050dba6a2022c32d9a6794631b1979978bfb890 Nov 03 19:57:14.833785 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.833745 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bn5rk" event={"ID":"e331a233-4c29-4910-8bca-06f847a906d6","Type":"ContainerStarted","Data":"cb1a49696ae633bad18ea1aa26993acc21f84938a10d224abb35d31bd6a90cb4"} Nov 03 19:57:14.834993 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.834970 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nbjzs" event={"ID":"865485da-2a04-435a-8b33-18beb925bf8e","Type":"ContainerStarted","Data":"28176f64d69b50e7bcf1bc6395afa7958a236efcc1d0634aefba5e2a97f04fd8"} Nov 03 19:57:14.837557 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.837528 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6tmzf" event={"ID":"224062e4-7b08-4d73-ba03-1789a9f591be","Type":"ContainerStarted","Data":"748d838f45e238781f66646edb83b4e0f0ed6efedb15aa579785d91ec2b78793"} Nov 03 19:57:14.839212 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.839189 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" event={"ID":"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2","Type":"ContainerStarted","Data":"4ab45c27df13c2a2bbdefc753a57a9bea7570fed7b268a82222dbcbc09bb709f"} Nov 03 19:57:14.839686 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.839657 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" event={"ID":"655da7d5-abe0-4c81-98a0-c4e2bf21b0a2","Type":"ContainerStarted","Data":"787cd666e097c65da9bae7838050dba6a2022c32d9a6794631b1979978bfb890"} Nov 03 19:57:14.839777 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.839721 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:14.845525 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.845423 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rwwdh" event={"ID":"dac45c98-1ae5-4137-bc74-943c81ac477b","Type":"ContainerStarted","Data":"eebfaf07f94d78ef5d26e91397e4e1aaf8cd9c3bb0e4a4e5421b3e7442046c8c"} Nov 03 19:57:14.882008 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:14.881956 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" podStartSLOduration=1.881943098 podStartE2EDuration="1.881943098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:14.881443821 +0000 UTC m=+13.257040572" watchObservedRunningTime="2026-11-03 19:57:14.881943098 +0000 UTC m=+13.257539839" Nov 03 19:57:15.852049 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:15.852023 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rwwdh" event={"ID":"dac45c98-1ae5-4137-bc74-943c81ac477b","Type":"ContainerStarted","Data":"05af21d594e8e5bce0152dc758726f0df86fdac7250f6191bc0cb42ebbc11736"} Nov 03 19:57:15.852382 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:15.852069 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rwwdh" Nov 03 19:57:15.857671 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:15.857648 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rwwdh" Nov 03 19:57:15.869661 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:15.869616 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-rwwdh" podStartSLOduration=1.726422788 podStartE2EDuration="2.869602371s" totalImagesPullingTime="1.143179583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:15.869448882 +0000 UTC m=+14.245045630" watchObservedRunningTime="2026-11-03 19:57:15.869602371 +0000 UTC m=+14.245199114" Nov 03 19:57:16.809432 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:16.809381 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5zxlx" Nov 03 19:57:16.858547 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:16.858483 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bn5rk" event={"ID":"e331a233-4c29-4910-8bca-06f847a906d6","Type":"ContainerStarted","Data":"47d380a003fbc041e4ea05bdef4779e3e07312578f26b3f7edd31b790b8f22e1"} Nov 03 19:57:16.885448 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:16.885370 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-bn5rk" podStartSLOduration=2.529008217 podStartE2EDuration="3.885357316s" totalImagesPullingTime="1.356349099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:16.884702524 +0000 UTC m=+15.260299274" watchObservedRunningTime="2026-11-03 19:57:16.885357316 +0000 UTC m=+15.260954063" Nov 03 19:57:21.163879 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.163755 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-62hrj"] Nov 03 19:57:21.170226 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.169697 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.246358 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.246323 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 19:57:21.246358 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.246347 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fksvw\"" Nov 03 19:57:21.246358 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.246367 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 19:57:21.256849 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.256446 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 19:57:21.256849 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.256600 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 19:57:21.256849 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.256684 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 19:57:21.256849 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.256821 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 19:57:21.276578 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.276547 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-tls\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.276749 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.276662 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-textfile\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.276749 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.276705 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.276749 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.276740 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl28f\" (UniqueName: \"kubernetes.io/projected/b09564c7-0bed-4d99-8cec-39a50dd69999-kube-api-access-fl28f\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.276909 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.276816 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-accelerators-collector-config\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.276962 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.276927 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b09564c7-0bed-4d99-8cec-39a50dd69999-sys\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.277011 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.276964 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b09564c7-0bed-4d99-8cec-39a50dd69999-root\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.277011 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.276991 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-wtmp\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.277111 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.277010 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b09564c7-0bed-4d99-8cec-39a50dd69999-metrics-client-ca\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.378243 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.378199 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl28f\" (UniqueName: \"kubernetes.io/projected/b09564c7-0bed-4d99-8cec-39a50dd69999-kube-api-access-fl28f\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.378439 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.378250 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-accelerators-collector-config\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.378439 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.378275 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b09564c7-0bed-4d99-8cec-39a50dd69999-sys\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.378439 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.378306 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b09564c7-0bed-4d99-8cec-39a50dd69999-root\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.378439 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.378325 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-wtmp\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.378439 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.378350 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b09564c7-0bed-4d99-8cec-39a50dd69999-metrics-client-ca\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.378439 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.378369 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-tls\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.378439 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.378409 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b09564c7-0bed-4d99-8cec-39a50dd69999-sys\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.378439 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.378430 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-textfile\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.378858 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.378470 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.378858 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.378641 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-wtmp\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.378858 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.378691 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b09564c7-0bed-4d99-8cec-39a50dd69999-root\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.379001 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.378981 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-accelerators-collector-config\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.379075 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.379053 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b09564c7-0bed-4d99-8cec-39a50dd69999-metrics-client-ca\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.379117 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.379106 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-textfile\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.381144 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.381121 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.381491 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.381469 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b09564c7-0bed-4d99-8cec-39a50dd69999-node-exporter-tls\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.390729 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.390700 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl28f\" (UniqueName: \"kubernetes.io/projected/b09564c7-0bed-4d99-8cec-39a50dd69999-kube-api-access-fl28f\") pod \"node-exporter-62hrj\" (UID: \"b09564c7-0bed-4d99-8cec-39a50dd69999\") " pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.486668 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.486588 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-62hrj" Nov 03 19:57:21.498622 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:21.498589 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09564c7_0bed_4d99_8cec_39a50dd69999.slice/crio-bf43664bd2b1f9ba55f6879bade7110599066187aae01aa3632a313d227c1d1e WatchSource:0}: Error finding container bf43664bd2b1f9ba55f6879bade7110599066187aae01aa3632a313d227c1d1e: Status 404 returned error can't find the container with id bf43664bd2b1f9ba55f6879bade7110599066187aae01aa3632a313d227c1d1e Nov 03 19:57:21.878191 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:21.878153 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-62hrj" event={"ID":"b09564c7-0bed-4d99-8cec-39a50dd69999","Type":"ContainerStarted","Data":"bf43664bd2b1f9ba55f6879bade7110599066187aae01aa3632a313d227c1d1e"} Nov 03 19:57:22.883380 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:22.883334 4457 generic.go:356] "Generic (PLEG): container finished" podID="b09564c7-0bed-4d99-8cec-39a50dd69999" containerID="950ede609b67fb60a10443065167ce7d920d134b23d19d101aa120a74a290c4b" exitCode=0 Nov 03 19:57:22.883834 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:22.883410 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-62hrj" event={"ID":"b09564c7-0bed-4d99-8cec-39a50dd69999","Type":"ContainerDied","Data":"950ede609b67fb60a10443065167ce7d920d134b23d19d101aa120a74a290c4b"} Nov 03 19:57:23.272902 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.272803 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2"] Nov 03 19:57:23.277023 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.276990 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.281882 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.281852 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 03 19:57:23.282039 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.281858 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 03 19:57:23.282209 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.282155 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d8qtvc450h3fg\"" Nov 03 19:57:23.282209 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.282176 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 03 19:57:23.282328 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.282247 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 03 19:57:23.283065 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.283029 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j5gvn\"" Nov 03 19:57:23.283065 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.283029 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 03 19:57:23.290626 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.290604 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2"] Nov 03 19:57:23.395560 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.395522 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-tls\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.395730 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.395574 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.395730 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.395609 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.395730 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.395699 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94288\" (UniqueName: \"kubernetes.io/projected/aafb6538-86d0-4d51-9afe-c4c49e749d1b-kube-api-access-94288\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.395894 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.395790 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aafb6538-86d0-4d51-9afe-c4c49e749d1b-metrics-client-ca\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.395894 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.395817 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.395894 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.395835 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-grpc-tls\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.395995 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.395908 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aafb6538-86d0-4d51-9afe-c4c49e749d1b-tmp\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.396062 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.396048 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.496893 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.496843 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.496893 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.496896 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.497148 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.496925 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94288\" (UniqueName: \"kubernetes.io/projected/aafb6538-86d0-4d51-9afe-c4c49e749d1b-kube-api-access-94288\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.497148 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.496946 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aafb6538-86d0-4d51-9afe-c4c49e749d1b-metrics-client-ca\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.497148 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.496964 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.497148 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.496979 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-grpc-tls\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.497330 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.497287 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aafb6538-86d0-4d51-9afe-c4c49e749d1b-tmp\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.497418 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.497377 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.497489 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.497447 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-tls\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.497837 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.497800 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aafb6538-86d0-4d51-9afe-c4c49e749d1b-tmp\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.497930 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.497881 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aafb6538-86d0-4d51-9afe-c4c49e749d1b-metrics-client-ca\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.499857 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.499825 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-tls\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.499958 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.499929 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-grpc-tls\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.500015 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.499936 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.500050 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.500035 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.500114 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.500095 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.500440 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.500421 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aafb6538-86d0-4d51-9afe-c4c49e749d1b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.509583 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.509564 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94288\" (UniqueName: \"kubernetes.io/projected/aafb6538-86d0-4d51-9afe-c4c49e749d1b-kube-api-access-94288\") pod \"thanos-querier-7bb8966dcb-xfqv2\" (UID: \"aafb6538-86d0-4d51-9afe-c4c49e749d1b\") " pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.603715 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.603676 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:23.889324 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.889244 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-62hrj" event={"ID":"b09564c7-0bed-4d99-8cec-39a50dd69999","Type":"ContainerStarted","Data":"2054ab2cbd10e8a9143af0891e78e0809353cbe231dbe7b40cb7fb406a25e42a"} Nov 03 19:57:23.889324 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.889298 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-62hrj" event={"ID":"b09564c7-0bed-4d99-8cec-39a50dd69999","Type":"ContainerStarted","Data":"181441c43e048f8642ea9baf8e101965cfb82f47ba7099b6902ed6f552c8ee96"} Nov 03 19:57:23.895587 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.895539 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2"] Nov 03 19:57:23.898618 ip-10-0-132-25 kubenswrapper[4457]: W1103 19:57:23.898586 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafb6538_86d0_4d51_9afe_c4c49e749d1b.slice/crio-b8131c9bf78d98f63d9b800236a337fdeb0fa8f2510be1033bb35f62a4b64578 WatchSource:0}: Error finding container b8131c9bf78d98f63d9b800236a337fdeb0fa8f2510be1033bb35f62a4b64578: Status 404 returned error can't find the container with id b8131c9bf78d98f63d9b800236a337fdeb0fa8f2510be1033bb35f62a4b64578 Nov 03 19:57:23.913252 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:23.913184 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-62hrj" podStartSLOduration=1.832160206 podStartE2EDuration="2.913169226s" totalImagesPullingTime="1.012718572s" totalInitContainerRuntime="68.290448ms" isStatefulPod=true podCreationTimestamp="2026-11-03 19:57:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:23.912531952 +0000 UTC m=+22.288128701" watchObservedRunningTime="2026-11-03 19:57:23.913169226 +0000 UTC m=+22.288765967" Nov 03 19:57:24.895716 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:24.895585 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" event={"ID":"aafb6538-86d0-4d51-9afe-c4c49e749d1b","Type":"ContainerStarted","Data":"b8131c9bf78d98f63d9b800236a337fdeb0fa8f2510be1033bb35f62a4b64578"} Nov 03 19:57:25.238272 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:25.238197 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc9nt" Nov 03 19:57:26.906141 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:26.906101 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" event={"ID":"aafb6538-86d0-4d51-9afe-c4c49e749d1b","Type":"ContainerStarted","Data":"68585bc88d044b06b26ae5e5b14307f9266dc3a7aff2b7afd7b2fe326682b910"} Nov 03 19:57:26.906700 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:26.906188 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" event={"ID":"aafb6538-86d0-4d51-9afe-c4c49e749d1b","Type":"ContainerStarted","Data":"ae7effab0540c56ccb76fcc9d2c737e4b434ddd94acb7cf17550beff374beeee"} Nov 03 19:57:26.906700 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:26.906231 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" event={"ID":"aafb6538-86d0-4d51-9afe-c4c49e749d1b","Type":"ContainerStarted","Data":"77a10e26ea012a6079ab248731ef2f081f3c96903523a6d5bb1c331f89fcfc56"} Nov 03 19:57:27.914802 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:27.914767 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" event={"ID":"aafb6538-86d0-4d51-9afe-c4c49e749d1b","Type":"ContainerStarted","Data":"db6741f502117ae1199adfae13fea56ad9ba7095e04da97a713e7db75940eeb6"} Nov 03 19:57:27.915280 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:27.914845 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" event={"ID":"aafb6538-86d0-4d51-9afe-c4c49e749d1b","Type":"ContainerStarted","Data":"b4fa51f8b50b430c0479627f3ab11d21b68f92c661d4a6da3e65e3ca7aacbc2a"} Nov 03 19:57:27.915280 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:27.914889 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" event={"ID":"aafb6538-86d0-4d51-9afe-c4c49e749d1b","Type":"ContainerStarted","Data":"4469e74e4a54bf2614e4538807ab382acf671e7298f7f02ac38d753fc037def0"} Nov 03 19:57:27.915280 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:27.915033 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:27.945361 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:27.945298 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" podStartSLOduration=1.6848269569999998 podStartE2EDuration="4.945278976s" totalImagesPullingTime="3.260452019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:27.943198158 +0000 UTC m=+26.318794920" watchObservedRunningTime="2026-11-03 19:57:27.945278976 +0000 UTC m=+26.320875723" Nov 03 19:57:32.937808 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:32.937756 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nbjzs" event={"ID":"865485da-2a04-435a-8b33-18beb925bf8e","Type":"ContainerStarted","Data":"d69a93cedd3b7567e9eb173e1f81d6a61ee7f5f7fc4a01cf416d4e371d1b0ae0"} Nov 03 19:57:32.938522 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:32.937866 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-nbjzs" Nov 03 19:57:32.960525 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:32.960463 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-nbjzs" podStartSLOduration=2.279965649 podStartE2EDuration="19.960444452s" totalImagesPullingTime="17.680478803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 19:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 19:57:32.958496418 +0000 UTC m=+31.334093166" watchObservedRunningTime="2026-11-03 19:57:32.960444452 +0000 UTC m=+31.336041200" Nov 03 19:57:32.961488 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:32.961461 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-nbjzs" Nov 03 19:57:33.793277 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:33.793238 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-h4zrs" Nov 03 19:57:33.927254 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:33.927227 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7bb8966dcb-xfqv2" Nov 03 19:57:35.857062 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:35.857030 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67895ffd5b-vwr6d" Nov 03 19:57:36.148619 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:57:36.148544 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 19:57:39.683989 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:39.683954 4457 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ktt5_642aaac2-4dc2-4800-bc55-c70730884400/konnectivity-agent/0.log" limit=1024 Nov 03 19:57:39.789403 ip-10-0-132-25 kubenswrapper[4457]: I1103 19:57:39.789346 4457 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9ktt5_642aaac2-4dc2-4800-bc55-c70730884400/konnectivity-agent/0.log" limit=1024 Nov 03 19:58:06.143851 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:58:06.143812 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 19:58:36.142905 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:58:36.142870 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 19:59:06.143413 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:59:06.143354 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 19:59:36.143193 ip-10-0-132-25 kubenswrapper[4457]: E1103 19:59:36.143109 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:00:06.144126 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:00:06.144078 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:00:36.144351 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:00:36.144311 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:01:06.143514 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:01:06.143477 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:01:36.143556 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:01:36.143522 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:02:01.757093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:02:01.757066 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:02:01.757708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:02:01.757684 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:02:06.143382 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:02:06.143344 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:02:36.143777 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:02:36.143701 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:03:06.144813 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:03:06.144774 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:03:18.608746 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.608710 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7"] Nov 03 20:03:18.611141 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.611118 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" Nov 03 20:03:18.616841 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.616819 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 20:03:18.616940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.616910 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 20:03:18.617044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.617025 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-ft92x\"" Nov 03 20:03:18.635377 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.635355 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7"] Nov 03 20:03:18.649147 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.649125 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8"] Nov 03 20:03:18.651334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.651311 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" Nov 03 20:03:18.661763 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.661740 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8"] Nov 03 20:03:18.688452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.688427 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdwz7\" (UniqueName: \"kubernetes.io/projected/795d21d6-9f81-44fd-a0e0-743753d41eb5-kube-api-access-qdwz7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8\" (UID: \"795d21d6-9f81-44fd-a0e0-743753d41eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" Nov 03 20:03:18.688567 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.688481 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8p2\" (UniqueName: \"kubernetes.io/projected/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-kube-api-access-ks8p2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7\" (UID: \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" Nov 03 20:03:18.688607 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.688581 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7\" (UID: \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" Nov 03 20:03:18.688665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.688651 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8\" (UID: \"795d21d6-9f81-44fd-a0e0-743753d41eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" Nov 03 20:03:18.688704 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.688682 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7\" (UID: \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" Nov 03 20:03:18.688746 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.688712 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8\" (UID: \"795d21d6-9f81-44fd-a0e0-743753d41eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" Nov 03 20:03:18.717540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.717517 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-qp6n5\"" Nov 03 20:03:18.789111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.789081 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8p2\" (UniqueName: \"kubernetes.io/projected/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-kube-api-access-ks8p2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7\" (UID: \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" Nov 03 20:03:18.789300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.789117 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7\" (UID: \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" Nov 03 20:03:18.789300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.789145 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8\" (UID: \"795d21d6-9f81-44fd-a0e0-743753d41eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" Nov 03 20:03:18.789300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.789171 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7\" (UID: \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" Nov 03 20:03:18.789300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.789207 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8\" (UID: \"795d21d6-9f81-44fd-a0e0-743753d41eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" Nov 03 20:03:18.789300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.789237 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdwz7\" (UniqueName: \"kubernetes.io/projected/795d21d6-9f81-44fd-a0e0-743753d41eb5-kube-api-access-qdwz7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8\" (UID: \"795d21d6-9f81-44fd-a0e0-743753d41eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" Nov 03 20:03:18.789683 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.789664 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7\" (UID: \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" Nov 03 20:03:18.789739 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.789675 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8\" (UID: \"795d21d6-9f81-44fd-a0e0-743753d41eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" Nov 03 20:03:18.789775 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.789738 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7\" (UID: \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" Nov 03 20:03:18.789775 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.789753 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8\" (UID: \"795d21d6-9f81-44fd-a0e0-743753d41eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" Nov 03 20:03:18.804452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.804431 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8p2\" (UniqueName: \"kubernetes.io/projected/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-kube-api-access-ks8p2\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7\" (UID: \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" Nov 03 20:03:18.804685 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.804663 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdwz7\" (UniqueName: \"kubernetes.io/projected/795d21d6-9f81-44fd-a0e0-743753d41eb5-kube-api-access-qdwz7\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8\" (UID: \"795d21d6-9f81-44fd-a0e0-743753d41eb5\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" Nov 03 20:03:18.923887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.923802 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" Nov 03 20:03:18.965024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:18.964990 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" Nov 03 20:03:19.200343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:19.200318 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7"] Nov 03 20:03:19.202717 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:03:19.202694 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed9e2be_fd53_4ae5_a3f2_c329632e79dc.slice/crio-15749cffb93a77ffc2ceb9eeca07001cf7c7de1fb3fd57e77bf440aec04d4823 WatchSource:0}: Error finding container 15749cffb93a77ffc2ceb9eeca07001cf7c7de1fb3fd57e77bf440aec04d4823: Status 404 returned error can't find the container with id 15749cffb93a77ffc2ceb9eeca07001cf7c7de1fb3fd57e77bf440aec04d4823 Nov 03 20:03:19.204584 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:19.204560 4457 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:03:19.244569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:19.244541 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8"] Nov 03 20:03:19.247116 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:03:19.247091 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795d21d6_9f81_44fd_a0e0_743753d41eb5.slice/crio-58f645db1ab6cecef7b7b1f8e837b899cd27ecc2a994f957782d9afda6f3b17c WatchSource:0}: Error finding container 58f645db1ab6cecef7b7b1f8e837b899cd27ecc2a994f957782d9afda6f3b17c: Status 404 returned error can't find the container with id 58f645db1ab6cecef7b7b1f8e837b899cd27ecc2a994f957782d9afda6f3b17c Nov 03 20:03:20.054970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:20.054925 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" event={"ID":"795d21d6-9f81-44fd-a0e0-743753d41eb5","Type":"ContainerStarted","Data":"58f645db1ab6cecef7b7b1f8e837b899cd27ecc2a994f957782d9afda6f3b17c"} Nov 03 20:03:20.056314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:20.056278 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" event={"ID":"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc","Type":"ContainerStarted","Data":"15749cffb93a77ffc2ceb9eeca07001cf7c7de1fb3fd57e77bf440aec04d4823"} Nov 03 20:03:21.428882 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.428850 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t"] Nov 03 20:03:21.431545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.431526 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" Nov 03 20:03:21.436430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.436412 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-zk48c\"" Nov 03 20:03:21.442098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.442071 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t"] Nov 03 20:03:21.505706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.505668 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64zlp\" (UniqueName: \"kubernetes.io/projected/59f1e9cb-4fba-42c6-a927-f48228db3119-kube-api-access-64zlp\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t\" (UID: \"59f1e9cb-4fba-42c6-a927-f48228db3119\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" Nov 03 20:03:21.506004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.505762 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t\" (UID: \"59f1e9cb-4fba-42c6-a927-f48228db3119\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" Nov 03 20:03:21.506004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.505798 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t\" (UID: \"59f1e9cb-4fba-42c6-a927-f48228db3119\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" Nov 03 20:03:21.607094 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.607049 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64zlp\" (UniqueName: \"kubernetes.io/projected/59f1e9cb-4fba-42c6-a927-f48228db3119-kube-api-access-64zlp\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t\" (UID: \"59f1e9cb-4fba-42c6-a927-f48228db3119\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" Nov 03 20:03:21.607287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.607217 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t\" (UID: \"59f1e9cb-4fba-42c6-a927-f48228db3119\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" Nov 03 20:03:21.607287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.607245 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t\" (UID: \"59f1e9cb-4fba-42c6-a927-f48228db3119\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" Nov 03 20:03:21.607850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.607799 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t\" (UID: \"59f1e9cb-4fba-42c6-a927-f48228db3119\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" Nov 03 20:03:21.607850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.607825 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t\" (UID: \"59f1e9cb-4fba-42c6-a927-f48228db3119\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" Nov 03 20:03:21.621006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.620975 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64zlp\" (UniqueName: \"kubernetes.io/projected/59f1e9cb-4fba-42c6-a927-f48228db3119-kube-api-access-64zlp\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t\" (UID: \"59f1e9cb-4fba-42c6-a927-f48228db3119\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" Nov 03 20:03:21.748331 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:21.748052 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" Nov 03 20:03:22.038477 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.038397 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t"] Nov 03 20:03:22.042565 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:03:22.042419 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f1e9cb_4fba_42c6_a927_f48228db3119.slice/crio-a05831772850f611ad5a271a9c5446ebf58827d6d36ef7cee4bfce215facc367 WatchSource:0}: Error finding container a05831772850f611ad5a271a9c5446ebf58827d6d36ef7cee4bfce215facc367: Status 404 returned error can't find the container with id a05831772850f611ad5a271a9c5446ebf58827d6d36ef7cee4bfce215facc367 Nov 03 20:03:22.065981 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.065947 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" event={"ID":"59f1e9cb-4fba-42c6-a927-f48228db3119","Type":"ContainerStarted","Data":"a05831772850f611ad5a271a9c5446ebf58827d6d36ef7cee4bfce215facc367"} Nov 03 20:03:22.669834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.669785 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm"] Nov 03 20:03:22.672742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.672717 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" Nov 03 20:03:22.678457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.677898 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-8vt82\"" Nov 03 20:03:22.685181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.685144 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm"] Nov 03 20:03:22.717207 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.716997 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjvx2\" (UniqueName: \"kubernetes.io/projected/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-kube-api-access-xjvx2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm\" (UID: \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" Nov 03 20:03:22.717207 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.717043 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm\" (UID: \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" Nov 03 20:03:22.717207 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.717085 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm\" (UID: \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" Nov 03 20:03:22.767125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.767090 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9"] Nov 03 20:03:22.769605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.769580 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" Nov 03 20:03:22.779413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.778199 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-nvcrg\"" Nov 03 20:03:22.787758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.782355 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9"] Nov 03 20:03:22.818259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.818229 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm\" (UID: \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" Nov 03 20:03:22.818437 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.818294 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scx62\" (UniqueName: \"kubernetes.io/projected/bac274e7-eed4-4ccc-b597-37581ed614e7-kube-api-access-scx62\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9\" (UID: \"bac274e7-eed4-4ccc-b597-37581ed614e7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" Nov 03 20:03:22.818504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.818479 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjvx2\" (UniqueName: \"kubernetes.io/projected/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-kube-api-access-xjvx2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm\" (UID: \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" Nov 03 20:03:22.818587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.818566 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9\" (UID: \"bac274e7-eed4-4ccc-b597-37581ed614e7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" Nov 03 20:03:22.818639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.818602 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm\" (UID: \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" Nov 03 20:03:22.818639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.818624 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9\" (UID: \"bac274e7-eed4-4ccc-b597-37581ed614e7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" Nov 03 20:03:22.818997 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.818980 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm\" (UID: \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" Nov 03 20:03:22.819114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.819094 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm\" (UID: \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" Nov 03 20:03:22.834230 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.834191 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjvx2\" (UniqueName: \"kubernetes.io/projected/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-kube-api-access-xjvx2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm\" (UID: \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" Nov 03 20:03:22.868348 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.868312 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c"] Nov 03 20:03:22.871429 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.871403 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" Nov 03 20:03:22.875450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.875427 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-72v9d\"" Nov 03 20:03:22.881483 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.880860 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c"] Nov 03 20:03:22.919546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.919510 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrvsd\" (UniqueName: \"kubernetes.io/projected/97e17443-fb36-43a0-a2d9-02b71b99d263-kube-api-access-rrvsd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c\" (UID: \"97e17443-fb36-43a0-a2d9-02b71b99d263\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" Nov 03 20:03:22.919712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.919561 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c\" (UID: \"97e17443-fb36-43a0-a2d9-02b71b99d263\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" Nov 03 20:03:22.919712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.919595 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scx62\" (UniqueName: \"kubernetes.io/projected/bac274e7-eed4-4ccc-b597-37581ed614e7-kube-api-access-scx62\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9\" (UID: \"bac274e7-eed4-4ccc-b597-37581ed614e7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" Nov 03 20:03:22.919712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.919620 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c\" (UID: \"97e17443-fb36-43a0-a2d9-02b71b99d263\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" Nov 03 20:03:22.919712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.919673 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9\" (UID: \"bac274e7-eed4-4ccc-b597-37581ed614e7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" Nov 03 20:03:22.919712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.919705 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9\" (UID: \"bac274e7-eed4-4ccc-b597-37581ed614e7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" Nov 03 20:03:22.920131 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.920073 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9\" (UID: \"bac274e7-eed4-4ccc-b597-37581ed614e7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" Nov 03 20:03:22.920348 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.920332 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9\" (UID: \"bac274e7-eed4-4ccc-b597-37581ed614e7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" Nov 03 20:03:22.933488 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.933453 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scx62\" (UniqueName: \"kubernetes.io/projected/bac274e7-eed4-4ccc-b597-37581ed614e7-kube-api-access-scx62\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9\" (UID: \"bac274e7-eed4-4ccc-b597-37581ed614e7\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" Nov 03 20:03:22.972438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.971530 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb"] Nov 03 20:03:22.975165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.975138 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" Nov 03 20:03:22.986984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.986933 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-xbxl5\"" Nov 03 20:03:22.988423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.988381 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb"] Nov 03 20:03:22.990783 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:22.990759 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" Nov 03 20:03:23.020803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.020769 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9jwv\" (UniqueName: \"kubernetes.io/projected/d7476a38-ebfa-48b2-a595-bf0d97d48707-kube-api-access-q9jwv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb\" (UID: \"d7476a38-ebfa-48b2-a595-bf0d97d48707\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" Nov 03 20:03:23.020945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.020834 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrvsd\" (UniqueName: \"kubernetes.io/projected/97e17443-fb36-43a0-a2d9-02b71b99d263-kube-api-access-rrvsd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c\" (UID: \"97e17443-fb36-43a0-a2d9-02b71b99d263\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" Nov 03 20:03:23.020945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.020863 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c\" (UID: \"97e17443-fb36-43a0-a2d9-02b71b99d263\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" Nov 03 20:03:23.020945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.020909 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c\" (UID: \"97e17443-fb36-43a0-a2d9-02b71b99d263\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" Nov 03 20:03:23.021114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.020949 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb\" (UID: \"d7476a38-ebfa-48b2-a595-bf0d97d48707\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" Nov 03 20:03:23.021114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.020981 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb\" (UID: \"d7476a38-ebfa-48b2-a595-bf0d97d48707\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" Nov 03 20:03:23.021741 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.021714 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c\" (UID: \"97e17443-fb36-43a0-a2d9-02b71b99d263\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" Nov 03 20:03:23.022002 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.021977 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c\" (UID: \"97e17443-fb36-43a0-a2d9-02b71b99d263\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" Nov 03 20:03:23.035094 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.035030 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrvsd\" (UniqueName: \"kubernetes.io/projected/97e17443-fb36-43a0-a2d9-02b71b99d263-kube-api-access-rrvsd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c\" (UID: \"97e17443-fb36-43a0-a2d9-02b71b99d263\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" Nov 03 20:03:23.091425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.091197 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" Nov 03 20:03:23.122529 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.121775 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb\" (UID: \"d7476a38-ebfa-48b2-a595-bf0d97d48707\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" Nov 03 20:03:23.122529 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.121892 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9jwv\" (UniqueName: \"kubernetes.io/projected/d7476a38-ebfa-48b2-a595-bf0d97d48707-kube-api-access-q9jwv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb\" (UID: \"d7476a38-ebfa-48b2-a595-bf0d97d48707\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" Nov 03 20:03:23.122529 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.121977 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb\" (UID: \"d7476a38-ebfa-48b2-a595-bf0d97d48707\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" Nov 03 20:03:23.122529 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.122471 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb\" (UID: \"d7476a38-ebfa-48b2-a595-bf0d97d48707\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" Nov 03 20:03:23.122827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.122641 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb\" (UID: \"d7476a38-ebfa-48b2-a595-bf0d97d48707\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" Nov 03 20:03:23.135589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.135531 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9jwv\" (UniqueName: \"kubernetes.io/projected/d7476a38-ebfa-48b2-a595-bf0d97d48707-kube-api-access-q9jwv\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb\" (UID: \"d7476a38-ebfa-48b2-a595-bf0d97d48707\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" Nov 03 20:03:23.195782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.195505 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" Nov 03 20:03:23.280457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.279842 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm"] Nov 03 20:03:23.284002 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:03:23.283939 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb669aa2f_f7ef_4d4a_92a8_fe81fc862d1d.slice/crio-2bf133d2d9de96db9afa4fb47bb85c45c038a2e86a9664bb41c7978bd5646a4a WatchSource:0}: Error finding container 2bf133d2d9de96db9afa4fb47bb85c45c038a2e86a9664bb41c7978bd5646a4a: Status 404 returned error can't find the container with id 2bf133d2d9de96db9afa4fb47bb85c45c038a2e86a9664bb41c7978bd5646a4a Nov 03 20:03:23.303825 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.303801 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" Nov 03 20:03:23.376354 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.376274 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9"] Nov 03 20:03:23.382229 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:03:23.382192 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbac274e7_eed4_4ccc_b597_37581ed614e7.slice/crio-ba3c6c2ac3d2fb7bd9083c1afad5eac431ea95bcfc53e72305996b6060718882 WatchSource:0}: Error finding container ba3c6c2ac3d2fb7bd9083c1afad5eac431ea95bcfc53e72305996b6060718882: Status 404 returned error can't find the container with id ba3c6c2ac3d2fb7bd9083c1afad5eac431ea95bcfc53e72305996b6060718882 Nov 03 20:03:23.494922 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.494511 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c"] Nov 03 20:03:23.496923 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:03:23.496881 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e17443_fb36_43a0_a2d9_02b71b99d263.slice/crio-d4d58de634c5f5e2f7c93f988d08dd815da3928fc26e2011b55674b93cddaf10 WatchSource:0}: Error finding container d4d58de634c5f5e2f7c93f988d08dd815da3928fc26e2011b55674b93cddaf10: Status 404 returned error can't find the container with id d4d58de634c5f5e2f7c93f988d08dd815da3928fc26e2011b55674b93cddaf10 Nov 03 20:03:23.597036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:23.596973 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb"] Nov 03 20:03:23.601954 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:03:23.601922 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7476a38_ebfa_48b2_a595_bf0d97d48707.slice/crio-b29b17d26e14a94b08e95d9cd9852bd9a6278008ff8925b8c2d3119be8dca0fa WatchSource:0}: Error finding container b29b17d26e14a94b08e95d9cd9852bd9a6278008ff8925b8c2d3119be8dca0fa: Status 404 returned error can't find the container with id b29b17d26e14a94b08e95d9cd9852bd9a6278008ff8925b8c2d3119be8dca0fa Nov 03 20:03:24.078550 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.078450 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" event={"ID":"bac274e7-eed4-4ccc-b597-37581ed614e7","Type":"ContainerStarted","Data":"ba3c6c2ac3d2fb7bd9083c1afad5eac431ea95bcfc53e72305996b6060718882"} Nov 03 20:03:24.080736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.080526 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" event={"ID":"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d","Type":"ContainerStarted","Data":"2bf133d2d9de96db9afa4fb47bb85c45c038a2e86a9664bb41c7978bd5646a4a"} Nov 03 20:03:24.082259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.082203 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" event={"ID":"d7476a38-ebfa-48b2-a595-bf0d97d48707","Type":"ContainerStarted","Data":"b29b17d26e14a94b08e95d9cd9852bd9a6278008ff8925b8c2d3119be8dca0fa"} Nov 03 20:03:24.083884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.083829 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" event={"ID":"97e17443-fb36-43a0-a2d9-02b71b99d263","Type":"ContainerStarted","Data":"d4d58de634c5f5e2f7c93f988d08dd815da3928fc26e2011b55674b93cddaf10"} Nov 03 20:03:24.437894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.437812 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv"] Nov 03 20:03:24.442399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.442366 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" Nov 03 20:03:24.468555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.468527 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv"] Nov 03 20:03:24.468689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.468591 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd"] Nov 03 20:03:24.473254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.472549 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.495226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.494723 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 20:03:24.495958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.495637 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 20:03:24.495958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.495720 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 20:03:24.495958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.495925 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-gwd2g\"" Nov 03 20:03:24.498341 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.498275 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd"] Nov 03 20:03:24.536272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.536239 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.536272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.536278 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvj6c\" (UniqueName: \"kubernetes.io/projected/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-kube-api-access-dvj6c\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.536544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.536303 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv\" (UID: \"2a47a585-8e30-4ec0-b34d-48477530d0ad\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" Nov 03 20:03:24.536544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.536325 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.536544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.536342 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.536544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.536362 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwsbx\" (UniqueName: \"kubernetes.io/projected/2a47a585-8e30-4ec0-b34d-48477530d0ad-kube-api-access-gwsbx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv\" (UID: \"2a47a585-8e30-4ec0-b34d-48477530d0ad\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" Nov 03 20:03:24.536544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.536517 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.536813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.536556 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.536813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.536585 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv\" (UID: \"2a47a585-8e30-4ec0-b34d-48477530d0ad\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" Nov 03 20:03:24.536813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.536609 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.536813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.536636 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.536813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.536664 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.546466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.546037 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59"] Nov 03 20:03:24.551604 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.551581 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.565452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.565363 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59"] Nov 03 20:03:24.581492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.581467 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-gfnd6\"" Nov 03 20:03:24.637867 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.637830 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.638063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.637891 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.638063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.637929 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv\" (UID: \"2a47a585-8e30-4ec0-b34d-48477530d0ad\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" Nov 03 20:03:24.638063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.637955 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.638063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.637999 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.638063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638027 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.638063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638060 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.638362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638098 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvj6c\" (UniqueName: \"kubernetes.io/projected/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-kube-api-access-dvj6c\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.638362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638132 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.638362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638158 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d281fa-6f07-44cf-a71f-2de479e23415-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.638362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638189 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djld5\" (UniqueName: \"kubernetes.io/projected/80d281fa-6f07-44cf-a71f-2de479e23415-kube-api-access-djld5\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.638362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638221 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv\" (UID: \"2a47a585-8e30-4ec0-b34d-48477530d0ad\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" Nov 03 20:03:24.638362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638270 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.638362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638304 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d281fa-6f07-44cf-a71f-2de479e23415-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.638362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638339 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.638734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638367 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.638734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638407 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.638734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638446 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d281fa-6f07-44cf-a71f-2de479e23415-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.638734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638477 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwsbx\" (UniqueName: \"kubernetes.io/projected/2a47a585-8e30-4ec0-b34d-48477530d0ad-kube-api-access-gwsbx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv\" (UID: \"2a47a585-8e30-4ec0-b34d-48477530d0ad\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" Nov 03 20:03:24.638734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638504 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.638734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.638543 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.639300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.639273 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.641792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.639602 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv\" (UID: \"2a47a585-8e30-4ec0-b34d-48477530d0ad\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" Nov 03 20:03:24.641792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.640201 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv\" (UID: \"2a47a585-8e30-4ec0-b34d-48477530d0ad\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" Nov 03 20:03:24.641792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.640493 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.641792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.641138 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.641792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.641755 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.642112 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.641843 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.677403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.677285 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvj6c\" (UniqueName: \"kubernetes.io/projected/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-kube-api-access-dvj6c\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.678192 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.678138 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.679024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.678975 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.679295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.679269 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93f7f2c5-96a8-4ccc-ad9b-3cc648f85802-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-p7lrd\" (UID: \"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.681750 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.681717 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwsbx\" (UniqueName: \"kubernetes.io/projected/2a47a585-8e30-4ec0-b34d-48477530d0ad-kube-api-access-gwsbx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv\" (UID: \"2a47a585-8e30-4ec0-b34d-48477530d0ad\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.739127 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d281fa-6f07-44cf-a71f-2de479e23415-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.739173 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.739210 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.739290 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.739314 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d281fa-6f07-44cf-a71f-2de479e23415-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.739342 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djld5\" (UniqueName: \"kubernetes.io/projected/80d281fa-6f07-44cf-a71f-2de479e23415-kube-api-access-djld5\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.739376 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.739425 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d281fa-6f07-44cf-a71f-2de479e23415-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.739456 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.739865 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.740466 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.740695 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.741877 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d281fa-6f07-44cf-a71f-2de479e23415-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.742119 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.744086 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.744022 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d281fa-6f07-44cf-a71f-2de479e23415-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.745811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.745735 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d281fa-6f07-44cf-a71f-2de479e23415-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.752645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.752599 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d281fa-6f07-44cf-a71f-2de479e23415-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.758834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.758811 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djld5\" (UniqueName: \"kubernetes.io/projected/80d281fa-6f07-44cf-a71f-2de479e23415-kube-api-access-djld5\") pod \"openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59\" (UID: \"80d281fa-6f07-44cf-a71f-2de479e23415\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.777778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.777512 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" Nov 03 20:03:24.810704 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.810435 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:24.870056 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.869742 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:24.925804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.925772 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr"] Nov 03 20:03:24.931104 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.931079 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" Nov 03 20:03:24.942265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.942243 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr"] Nov 03 20:03:24.942833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.942657 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr\" (UID: \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" Nov 03 20:03:24.942833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.942715 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qlnk\" (UniqueName: \"kubernetes.io/projected/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-kube-api-access-7qlnk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr\" (UID: \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" Nov 03 20:03:24.942833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:24.942742 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr\" (UID: \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" Nov 03 20:03:25.044898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:25.043713 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qlnk\" (UniqueName: \"kubernetes.io/projected/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-kube-api-access-7qlnk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr\" (UID: \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" Nov 03 20:03:25.044898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:25.043760 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr\" (UID: \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" Nov 03 20:03:25.044898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:25.043851 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr\" (UID: \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" Nov 03 20:03:25.044898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:25.044264 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr\" (UID: \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" Nov 03 20:03:25.044898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:25.044850 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr\" (UID: \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" Nov 03 20:03:25.059999 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:25.059941 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qlnk\" (UniqueName: \"kubernetes.io/projected/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-kube-api-access-7qlnk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr\" (UID: \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" Nov 03 20:03:25.071690 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:25.071616 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv"] Nov 03 20:03:25.089623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:25.089578 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" event={"ID":"2a47a585-8e30-4ec0-b34d-48477530d0ad","Type":"ContainerStarted","Data":"ab8e222f24a47ab04778debfb77e9507208a7b4ca3c48d76fd4f89345dde89dd"} Nov 03 20:03:25.129375 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:25.129016 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd"] Nov 03 20:03:25.243189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:25.243137 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59"] Nov 03 20:03:25.255624 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:03:25.255574 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d281fa_6f07_44cf_a71f_2de479e23415.slice/crio-babe680c21f5ed93b3b4ec49bc47c11436137ed3792f8c4cb432495da0009969 WatchSource:0}: Error finding container babe680c21f5ed93b3b4ec49bc47c11436137ed3792f8c4cb432495da0009969: Status 404 returned error can't find the container with id babe680c21f5ed93b3b4ec49bc47c11436137ed3792f8c4cb432495da0009969 Nov 03 20:03:25.272181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:25.272153 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" Nov 03 20:03:25.587411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:25.587268 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr"] Nov 03 20:03:25.593335 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:03:25.593300 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb95a383_6a27_491e_b9e2_67c36bdeaaf0.slice/crio-c51bef9996c080f335976a2f44040e591b81c62b8820860c3ab2ced49852ee64 WatchSource:0}: Error finding container c51bef9996c080f335976a2f44040e591b81c62b8820860c3ab2ced49852ee64: Status 404 returned error can't find the container with id c51bef9996c080f335976a2f44040e591b81c62b8820860c3ab2ced49852ee64 Nov 03 20:03:26.114534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:26.114428 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" event={"ID":"eb95a383-6a27-491e-b9e2-67c36bdeaaf0","Type":"ContainerStarted","Data":"c51bef9996c080f335976a2f44040e591b81c62b8820860c3ab2ced49852ee64"} Nov 03 20:03:26.135500 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:26.135399 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" event={"ID":"80d281fa-6f07-44cf-a71f-2de479e23415","Type":"ContainerStarted","Data":"babe680c21f5ed93b3b4ec49bc47c11436137ed3792f8c4cb432495da0009969"} Nov 03 20:03:26.150812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:26.150702 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" event={"ID":"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802","Type":"ContainerStarted","Data":"adf57392155b987c63de12840b8bed4867b1ed80eafba8deff5ba7cf434340cc"} Nov 03 20:03:35.832557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:35.832354 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:03:35.832557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:35.832456 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:03:35.832557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:35.832498 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:03:36.156958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:36.156244 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:03:36.156958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:36.156327 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:03:36.156958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:36.156359 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:03:36.162159 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:03:36.162125 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:03:36.253272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:36.253237 4457 generic.go:356] "Generic (PLEG): container finished" podID="5ed9e2be-fd53-4ae5-a3f2-c329632e79dc" containerID="e68bfb17b0f93607439a2219aa6ad26ceaddf3a4c9e94914d07b20966305c76a" exitCode=0 Nov 03 20:03:36.253535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:36.253340 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" event={"ID":"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc","Type":"ContainerDied","Data":"e68bfb17b0f93607439a2219aa6ad26ceaddf3a4c9e94914d07b20966305c76a"} Nov 03 20:03:36.256853 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:36.256819 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" event={"ID":"80d281fa-6f07-44cf-a71f-2de479e23415","Type":"ContainerStarted","Data":"8250d018eb1f4001ecb1ec7fd549e44cc0b8e0203f605929c43b62fae4e1dd72"} Nov 03 20:03:36.264943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:36.264201 4457 generic.go:356] "Generic (PLEG): container finished" podID="d7476a38-ebfa-48b2-a595-bf0d97d48707" containerID="ab4c0aaae6d245f150d1787eb446b6b93fb04192e53114ac3e8589f5d68eafb5" exitCode=0 Nov 03 20:03:36.264943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:36.264268 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" event={"ID":"d7476a38-ebfa-48b2-a595-bf0d97d48707","Type":"ContainerDied","Data":"ab4c0aaae6d245f150d1787eb446b6b93fb04192e53114ac3e8589f5d68eafb5"} Nov 03 20:03:36.301342 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:36.301233 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" podStartSLOduration=1.728363013 podStartE2EDuration="12.301213278s" totalImagesPullingTime="10.572850265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:36.30026338 +0000 UTC m=+394.675860142" watchObservedRunningTime="2026-11-03 20:03:36.301213278 +0000 UTC m=+394.676810026" Nov 03 20:03:36.872807 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:36.872758 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:36.880259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:36.880231 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:37.272594 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:37.272509 4457 generic.go:356] "Generic (PLEG): container finished" podID="bac274e7-eed4-4ccc-b597-37581ed614e7" containerID="b4c8eed96345b20da307ffd42e67bf19de5bbea52cdea17954507e682ae6dcc6" exitCode=0 Nov 03 20:03:37.272847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:37.272815 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" event={"ID":"bac274e7-eed4-4ccc-b597-37581ed614e7","Type":"ContainerDied","Data":"b4c8eed96345b20da307ffd42e67bf19de5bbea52cdea17954507e682ae6dcc6"} Nov 03 20:03:37.275223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:37.275193 4457 generic.go:356] "Generic (PLEG): container finished" podID="2a47a585-8e30-4ec0-b34d-48477530d0ad" containerID="7a1337819bc68e2d3e01ef1976232d3d061e28e1d4c4485467ec6fc620fc1369" exitCode=0 Nov 03 20:03:37.275314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:37.275294 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" event={"ID":"2a47a585-8e30-4ec0-b34d-48477530d0ad","Type":"ContainerDied","Data":"7a1337819bc68e2d3e01ef1976232d3d061e28e1d4c4485467ec6fc620fc1369"} Nov 03 20:03:37.279573 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:37.279438 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" event={"ID":"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802","Type":"ContainerStarted","Data":"1f41ec797c3adc0673604f6e6ebb6628f7b4cdb9537252d2a3d3baf32fe76223"} Nov 03 20:03:37.279573 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:37.279503 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:37.280803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:37.280780 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59" Nov 03 20:03:37.361036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:37.360966 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" podStartSLOduration=2.343009671 podStartE2EDuration="13.360944932s" totalImagesPullingTime="11.017935261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:03:37.360657751 +0000 UTC m=+395.736254497" watchObservedRunningTime="2026-11-03 20:03:37.360944932 +0000 UTC m=+395.736541681" Nov 03 20:03:37.811591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:37.811552 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:37.816981 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:37.816720 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:38.286115 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:38.286079 4457 generic.go:356] "Generic (PLEG): container finished" podID="795d21d6-9f81-44fd-a0e0-743753d41eb5" containerID="c9512b070f1e799dfb15ecc84abcfebdb39bcb0f410798dfd8f4a9fa72804b93" exitCode=0 Nov 03 20:03:38.288218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:38.287561 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" event={"ID":"795d21d6-9f81-44fd-a0e0-743753d41eb5","Type":"ContainerDied","Data":"c9512b070f1e799dfb15ecc84abcfebdb39bcb0f410798dfd8f4a9fa72804b93"} Nov 03 20:03:38.289980 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:38.289955 4457 generic.go:356] "Generic (PLEG): container finished" podID="eb95a383-6a27-491e-b9e2-67c36bdeaaf0" containerID="ac75106d685094e4f4af23a627d9f3cda3f5f4038b320d1a0afc08478a4f9f2c" exitCode=0 Nov 03 20:03:38.290345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:38.290326 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" event={"ID":"eb95a383-6a27-491e-b9e2-67c36bdeaaf0","Type":"ContainerDied","Data":"ac75106d685094e4f4af23a627d9f3cda3f5f4038b320d1a0afc08478a4f9f2c"} Nov 03 20:03:38.290566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:38.290518 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:38.292276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:38.292258 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:03:39.299089 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:39.299053 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" event={"ID":"d7476a38-ebfa-48b2-a595-bf0d97d48707","Type":"ContainerStarted","Data":"9aadb9e5ca58294ac7a8782384cfde700001f34e94976eb63354bf943a89139e"} Nov 03 20:03:39.304171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:39.304091 4457 generic.go:356] "Generic (PLEG): container finished" podID="59f1e9cb-4fba-42c6-a927-f48228db3119" containerID="e3f023c25810814aafc8b945a9de632a59d1d372efbeb23dc642efe87c4737a5" exitCode=0 Nov 03 20:03:39.304256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:39.304185 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" event={"ID":"59f1e9cb-4fba-42c6-a927-f48228db3119","Type":"ContainerDied","Data":"e3f023c25810814aafc8b945a9de632a59d1d372efbeb23dc642efe87c4737a5"} Nov 03 20:03:39.310305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:39.309993 4457 generic.go:356] "Generic (PLEG): container finished" podID="bac274e7-eed4-4ccc-b597-37581ed614e7" containerID="099ed1bd5de585e9143fb8f09062fc61eda97e546d5eeea53155a8671eef7719" exitCode=0 Nov 03 20:03:39.310305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:39.310075 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" event={"ID":"bac274e7-eed4-4ccc-b597-37581ed614e7","Type":"ContainerDied","Data":"099ed1bd5de585e9143fb8f09062fc61eda97e546d5eeea53155a8671eef7719"} Nov 03 20:03:40.323551 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:40.323477 4457 generic.go:356] "Generic (PLEG): container finished" podID="795d21d6-9f81-44fd-a0e0-743753d41eb5" containerID="eb383bb4b2d8cde165839f7b667b7b26deb4c8f62fce00753e2ccc494e454a8f" exitCode=0 Nov 03 20:03:40.323958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:40.323578 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" event={"ID":"795d21d6-9f81-44fd-a0e0-743753d41eb5","Type":"ContainerDied","Data":"eb383bb4b2d8cde165839f7b667b7b26deb4c8f62fce00753e2ccc494e454a8f"} Nov 03 20:03:40.326435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:40.326407 4457 generic.go:356] "Generic (PLEG): container finished" podID="eb95a383-6a27-491e-b9e2-67c36bdeaaf0" containerID="380e7a287a7ac8e47e146e35de7bd0cc9c3f8959305812e4903af78ffd79c8cc" exitCode=0 Nov 03 20:03:40.326563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:40.326511 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" event={"ID":"eb95a383-6a27-491e-b9e2-67c36bdeaaf0","Type":"ContainerDied","Data":"380e7a287a7ac8e47e146e35de7bd0cc9c3f8959305812e4903af78ffd79c8cc"} Nov 03 20:03:40.330775 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:40.330681 4457 generic.go:356] "Generic (PLEG): container finished" podID="d7476a38-ebfa-48b2-a595-bf0d97d48707" containerID="9aadb9e5ca58294ac7a8782384cfde700001f34e94976eb63354bf943a89139e" exitCode=0 Nov 03 20:03:40.330775 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:40.330767 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" event={"ID":"d7476a38-ebfa-48b2-a595-bf0d97d48707","Type":"ContainerDied","Data":"9aadb9e5ca58294ac7a8782384cfde700001f34e94976eb63354bf943a89139e"} Nov 03 20:03:40.333260 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:40.333235 4457 generic.go:356] "Generic (PLEG): container finished" podID="97e17443-fb36-43a0-a2d9-02b71b99d263" containerID="4428d2ff5df1b35a53dc25644b289590b29650d941551786c941b98b44686dd1" exitCode=0 Nov 03 20:03:40.333449 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:40.333428 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" event={"ID":"97e17443-fb36-43a0-a2d9-02b71b99d263","Type":"ContainerDied","Data":"4428d2ff5df1b35a53dc25644b289590b29650d941551786c941b98b44686dd1"} Nov 03 20:03:40.338519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:40.337253 4457 generic.go:356] "Generic (PLEG): container finished" podID="b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d" containerID="8d4c9192232af5ba57ee21b855025c6a485c2b44525c6ffe6ca25f859cfd1476" exitCode=0 Nov 03 20:03:40.338519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:40.337413 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" event={"ID":"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d","Type":"ContainerDied","Data":"8d4c9192232af5ba57ee21b855025c6a485c2b44525c6ffe6ca25f859cfd1476"} Nov 03 20:03:41.351745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:41.350630 4457 generic.go:356] "Generic (PLEG): container finished" podID="5ed9e2be-fd53-4ae5-a3f2-c329632e79dc" containerID="5a43a100578fec49a5c582ed1ea8e3886fb3074d659b3bae1c9a667409c3c3a1" exitCode=0 Nov 03 20:03:41.351745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:41.350750 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" event={"ID":"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc","Type":"ContainerDied","Data":"5a43a100578fec49a5c582ed1ea8e3886fb3074d659b3bae1c9a667409c3c3a1"} Nov 03 20:03:41.360749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:41.360721 4457 generic.go:356] "Generic (PLEG): container finished" podID="2a47a585-8e30-4ec0-b34d-48477530d0ad" containerID="545f821892273591af766a99b9d19ff3ff0b3131a1a732f333c9fbc639245c0c" exitCode=0 Nov 03 20:03:41.360897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:41.360823 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" event={"ID":"2a47a585-8e30-4ec0-b34d-48477530d0ad","Type":"ContainerDied","Data":"545f821892273591af766a99b9d19ff3ff0b3131a1a732f333c9fbc639245c0c"} Nov 03 20:03:42.374265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:42.374179 4457 generic.go:356] "Generic (PLEG): container finished" podID="b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d" containerID="0211d65c10f188286b8e704c67c5fb37e2b5f98f1dd76b6890597573c528e741" exitCode=0 Nov 03 20:03:42.374750 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:42.374325 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" event={"ID":"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d","Type":"ContainerDied","Data":"0211d65c10f188286b8e704c67c5fb37e2b5f98f1dd76b6890597573c528e741"} Nov 03 20:03:42.383297 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:42.383266 4457 generic.go:356] "Generic (PLEG): container finished" podID="97e17443-fb36-43a0-a2d9-02b71b99d263" containerID="fc5331bcba81ebeded8de2d831a5f3d0c1353347dbd3b3cec7fd76a9dafd7c30" exitCode=0 Nov 03 20:03:42.383425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:42.383337 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" event={"ID":"97e17443-fb36-43a0-a2d9-02b71b99d263","Type":"ContainerDied","Data":"fc5331bcba81ebeded8de2d831a5f3d0c1353347dbd3b3cec7fd76a9dafd7c30"} Nov 03 20:03:44.405847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:44.405553 4457 generic.go:356] "Generic (PLEG): container finished" podID="59f1e9cb-4fba-42c6-a927-f48228db3119" containerID="d61d8fbd70ee4eb008ea64803b1d580c82169db5a4e662b864a82a26b6b18368" exitCode=0 Nov 03 20:03:44.405847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:44.405646 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" event={"ID":"59f1e9cb-4fba-42c6-a927-f48228db3119","Type":"ContainerDied","Data":"d61d8fbd70ee4eb008ea64803b1d580c82169db5a4e662b864a82a26b6b18368"} Nov 03 20:03:57.545015 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:57.544965 4457 generic.go:356] "Generic (PLEG): container finished" podID="59f1e9cb-4fba-42c6-a927-f48228db3119" containerID="2fd1f4e042ec68b9c23e91b097dbd20028cd9c339ae100c4a4a0f5a6c7438185" exitCode=0 Nov 03 20:03:57.545530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:57.545063 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" event={"ID":"59f1e9cb-4fba-42c6-a927-f48228db3119","Type":"ContainerDied","Data":"2fd1f4e042ec68b9c23e91b097dbd20028cd9c339ae100c4a4a0f5a6c7438185"} Nov 03 20:03:57.551025 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:57.550723 4457 generic.go:356] "Generic (PLEG): container finished" podID="bac274e7-eed4-4ccc-b597-37581ed614e7" containerID="b82a67719fa35b45c773c60e0150664973802b1cf692d8f7b6af00aa943eb168" exitCode=0 Nov 03 20:03:57.551025 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:57.550843 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" event={"ID":"bac274e7-eed4-4ccc-b597-37581ed614e7","Type":"ContainerDied","Data":"b82a67719fa35b45c773c60e0150664973802b1cf692d8f7b6af00aa943eb168"} Nov 03 20:03:57.555693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:57.555419 4457 generic.go:356] "Generic (PLEG): container finished" podID="d7476a38-ebfa-48b2-a595-bf0d97d48707" containerID="6bcedb74087975c3e4d18d04b1f816d1fe265e6e4fac86514c5b5ea44ce14e8f" exitCode=0 Nov 03 20:03:57.555693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:57.555468 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" event={"ID":"d7476a38-ebfa-48b2-a595-bf0d97d48707","Type":"ContainerDied","Data":"6bcedb74087975c3e4d18d04b1f816d1fe265e6e4fac86514c5b5ea44ce14e8f"} Nov 03 20:03:58.565065 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.565026 4457 generic.go:356] "Generic (PLEG): container finished" podID="795d21d6-9f81-44fd-a0e0-743753d41eb5" containerID="b3d8a68e00d46672bfaa5ab901da2d745cce929a5a9111ad5a3b52c3ab283a20" exitCode=0 Nov 03 20:03:58.565828 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.565800 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" event={"ID":"795d21d6-9f81-44fd-a0e0-743753d41eb5","Type":"ContainerDied","Data":"b3d8a68e00d46672bfaa5ab901da2d745cce929a5a9111ad5a3b52c3ab283a20"} Nov 03 20:03:58.788703 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.788623 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" Nov 03 20:03:58.848412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.848364 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bac274e7-eed4-4ccc-b597-37581ed614e7-kube-api-access-scx62\" (UniqueName: \"kubernetes.io/projected/bac274e7-eed4-4ccc-b597-37581ed614e7-kube-api-access-scx62\") pod \"bac274e7-eed4-4ccc-b597-37581ed614e7\" (UID: \"bac274e7-eed4-4ccc-b597-37581ed614e7\") " Nov 03 20:03:58.848607 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.848446 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-util\" (UniqueName: \"kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-util\") pod \"bac274e7-eed4-4ccc-b597-37581ed614e7\" (UID: \"bac274e7-eed4-4ccc-b597-37581ed614e7\") " Nov 03 20:03:58.848607 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.848557 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-bundle\") pod \"bac274e7-eed4-4ccc-b597-37581ed614e7\" (UID: \"bac274e7-eed4-4ccc-b597-37581ed614e7\") " Nov 03 20:03:58.849966 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.849936 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-bundle" pod "bac274e7-eed4-4ccc-b597-37581ed614e7" (UID: "bac274e7-eed4-4ccc-b597-37581ed614e7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:58.853913 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.853608 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" Nov 03 20:03:58.854746 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.854719 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-util" pod "bac274e7-eed4-4ccc-b597-37581ed614e7" (UID: "bac274e7-eed4-4ccc-b597-37581ed614e7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:58.854960 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.854937 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bac274e7-eed4-4ccc-b597-37581ed614e7-kube-api-access-scx62" pod "bac274e7-eed4-4ccc-b597-37581ed614e7" (UID: "bac274e7-eed4-4ccc-b597-37581ed614e7"). InnerVolumeSpecName "kube-api-access-scx62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:58.872337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.871437 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" Nov 03 20:03:58.950757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.950013 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7476a38-ebfa-48b2-a595-bf0d97d48707-kube-api-access-q9jwv\" (UniqueName: \"kubernetes.io/projected/d7476a38-ebfa-48b2-a595-bf0d97d48707-kube-api-access-q9jwv\") pod \"d7476a38-ebfa-48b2-a595-bf0d97d48707\" (UID: \"d7476a38-ebfa-48b2-a595-bf0d97d48707\") " Nov 03 20:03:58.950757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.950087 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-bundle\") pod \"d7476a38-ebfa-48b2-a595-bf0d97d48707\" (UID: \"d7476a38-ebfa-48b2-a595-bf0d97d48707\") " Nov 03 20:03:58.950757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.950154 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-util\" (UniqueName: \"kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-util\") pod \"59f1e9cb-4fba-42c6-a927-f48228db3119\" (UID: \"59f1e9cb-4fba-42c6-a927-f48228db3119\") " Nov 03 20:03:58.950757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.950220 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-bundle\" (UniqueName: \"kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-bundle\") pod \"59f1e9cb-4fba-42c6-a927-f48228db3119\" (UID: \"59f1e9cb-4fba-42c6-a927-f48228db3119\") " Nov 03 20:03:58.950757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.950261 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f1e9cb-4fba-42c6-a927-f48228db3119-kube-api-access-64zlp\" (UniqueName: \"kubernetes.io/projected/59f1e9cb-4fba-42c6-a927-f48228db3119-kube-api-access-64zlp\") pod \"59f1e9cb-4fba-42c6-a927-f48228db3119\" (UID: \"59f1e9cb-4fba-42c6-a927-f48228db3119\") " Nov 03 20:03:58.950757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.950310 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-util\" (UniqueName: \"kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-util\") pod \"d7476a38-ebfa-48b2-a595-bf0d97d48707\" (UID: \"d7476a38-ebfa-48b2-a595-bf0d97d48707\") " Nov 03 20:03:58.950757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.950506 4457 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-bundle\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:58.950757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.950524 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scx62\" (UniqueName: \"kubernetes.io/projected/bac274e7-eed4-4ccc-b597-37581ed614e7-kube-api-access-scx62\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:58.950757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.950540 4457 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bac274e7-eed4-4ccc-b597-37581ed614e7-util\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:58.952009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.951970 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-bundle" pod "59f1e9cb-4fba-42c6-a927-f48228db3119" (UID: "59f1e9cb-4fba-42c6-a927-f48228db3119"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:58.952329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.952301 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-bundle" pod "d7476a38-ebfa-48b2-a595-bf0d97d48707" (UID: "d7476a38-ebfa-48b2-a595-bf0d97d48707"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:58.955602 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.955568 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f1e9cb-4fba-42c6-a927-f48228db3119-kube-api-access-64zlp" pod "59f1e9cb-4fba-42c6-a927-f48228db3119" (UID: "59f1e9cb-4fba-42c6-a927-f48228db3119"). InnerVolumeSpecName "kube-api-access-64zlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:58.956837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.956809 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7476a38-ebfa-48b2-a595-bf0d97d48707-kube-api-access-q9jwv" pod "d7476a38-ebfa-48b2-a595-bf0d97d48707" (UID: "d7476a38-ebfa-48b2-a595-bf0d97d48707"). InnerVolumeSpecName "kube-api-access-q9jwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:58.964567 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.964542 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-util" pod "59f1e9cb-4fba-42c6-a927-f48228db3119" (UID: "59f1e9cb-4fba-42c6-a927-f48228db3119"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:58.964744 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:58.964725 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-util" pod "d7476a38-ebfa-48b2-a595-bf0d97d48707" (UID: "d7476a38-ebfa-48b2-a595-bf0d97d48707"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:59.051578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.051541 4457 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-bundle\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:59.051578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.051577 4457 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-util\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:59.051847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.051591 4457 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/59f1e9cb-4fba-42c6-a927-f48228db3119-bundle\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:59.051847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.051609 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64zlp\" (UniqueName: \"kubernetes.io/projected/59f1e9cb-4fba-42c6-a927-f48228db3119-kube-api-access-64zlp\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:59.051847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.051623 4457 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d7476a38-ebfa-48b2-a595-bf0d97d48707-util\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:59.051847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.051636 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9jwv\" (UniqueName: \"kubernetes.io/projected/d7476a38-ebfa-48b2-a595-bf0d97d48707-kube-api-access-q9jwv\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:59.572331 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.572304 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" Nov 03 20:03:59.572331 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.572312 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6nj2q9" event={"ID":"bac274e7-eed4-4ccc-b597-37581ed614e7","Type":"ContainerDied","Data":"ba3c6c2ac3d2fb7bd9083c1afad5eac431ea95bcfc53e72305996b6060718882"} Nov 03 20:03:59.572794 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.572364 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba3c6c2ac3d2fb7bd9083c1afad5eac431ea95bcfc53e72305996b6060718882" Nov 03 20:03:59.574510 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.574487 4457 generic.go:356] "Generic (PLEG): container finished" podID="eb95a383-6a27-491e-b9e2-67c36bdeaaf0" containerID="7dc4dee457d1451e5243a9153dbcb94483c1778932696b0d6d1742d84a652f15" exitCode=0 Nov 03 20:03:59.574632 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.574526 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" event={"ID":"eb95a383-6a27-491e-b9e2-67c36bdeaaf0","Type":"ContainerDied","Data":"7dc4dee457d1451e5243a9153dbcb94483c1778932696b0d6d1742d84a652f15"} Nov 03 20:03:59.576436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.576415 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" Nov 03 20:03:59.576548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.576411 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cb4vzb" event={"ID":"d7476a38-ebfa-48b2-a595-bf0d97d48707","Type":"ContainerDied","Data":"b29b17d26e14a94b08e95d9cd9852bd9a6278008ff8925b8c2d3119be8dca0fa"} Nov 03 20:03:59.576548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.576527 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b29b17d26e14a94b08e95d9cd9852bd9a6278008ff8925b8c2d3119be8dca0fa" Nov 03 20:03:59.579191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.578577 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" event={"ID":"59f1e9cb-4fba-42c6-a927-f48228db3119","Type":"ContainerDied","Data":"a05831772850f611ad5a271a9c5446ebf58827d6d36ef7cee4bfce215facc367"} Nov 03 20:03:59.579191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.578618 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a05831772850f611ad5a271a9c5446ebf58827d6d36ef7cee4bfce215facc367" Nov 03 20:03:59.579191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.578622 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6447vb4t" Nov 03 20:03:59.704229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.704201 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" Nov 03 20:03:59.757535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.757508 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-util\" (UniqueName: \"kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-util\") pod \"795d21d6-9f81-44fd-a0e0-743753d41eb5\" (UID: \"795d21d6-9f81-44fd-a0e0-743753d41eb5\") " Nov 03 20:03:59.757678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.757544 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-bundle\") pod \"795d21d6-9f81-44fd-a0e0-743753d41eb5\" (UID: \"795d21d6-9f81-44fd-a0e0-743753d41eb5\") " Nov 03 20:03:59.757678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.757588 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/795d21d6-9f81-44fd-a0e0-743753d41eb5-kube-api-access-qdwz7\" (UniqueName: \"kubernetes.io/projected/795d21d6-9f81-44fd-a0e0-743753d41eb5-kube-api-access-qdwz7\") pod \"795d21d6-9f81-44fd-a0e0-743753d41eb5\" (UID: \"795d21d6-9f81-44fd-a0e0-743753d41eb5\") " Nov 03 20:03:59.758131 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.758095 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-bundle" pod "795d21d6-9f81-44fd-a0e0-743753d41eb5" (UID: "795d21d6-9f81-44fd-a0e0-743753d41eb5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:59.759626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.759604 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/795d21d6-9f81-44fd-a0e0-743753d41eb5-kube-api-access-qdwz7" pod "795d21d6-9f81-44fd-a0e0-743753d41eb5" (UID: "795d21d6-9f81-44fd-a0e0-743753d41eb5"). InnerVolumeSpecName "kube-api-access-qdwz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:03:59.763285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.763258 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-util" pod "795d21d6-9f81-44fd-a0e0-743753d41eb5" (UID: "795d21d6-9f81-44fd-a0e0-743753d41eb5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:03:59.858697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.858676 4457 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-util\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:59.858697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.858698 4457 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/795d21d6-9f81-44fd-a0e0-743753d41eb5-bundle\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:03:59.858827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:03:59.858709 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdwz7\" (UniqueName: \"kubernetes.io/projected/795d21d6-9f81-44fd-a0e0-743753d41eb5-kube-api-access-qdwz7\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:00.585270 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.585240 4457 generic.go:356] "Generic (PLEG): container finished" podID="97e17443-fb36-43a0-a2d9-02b71b99d263" containerID="3834b37659e2c72d79355c6a9e018eda30924c3d2e646448b72612d507485ffa" exitCode=0 Nov 03 20:04:00.585752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.585313 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" event={"ID":"97e17443-fb36-43a0-a2d9-02b71b99d263","Type":"ContainerDied","Data":"3834b37659e2c72d79355c6a9e018eda30924c3d2e646448b72612d507485ffa"} Nov 03 20:04:00.587293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.587268 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" event={"ID":"2a47a585-8e30-4ec0-b34d-48477530d0ad","Type":"ContainerStarted","Data":"73d318a7d62039a867a27464c139f6e3acf4b83fb1e22b15d6aff78c13ebe2fb"} Nov 03 20:04:00.589294 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.589231 4457 generic.go:356] "Generic (PLEG): container finished" podID="b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d" containerID="d21bef21f448a972acedab3b26b8d3152f9b5e6e328f2af081ef36c3fa4d32ba" exitCode=0 Nov 03 20:04:00.589419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.589267 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" event={"ID":"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d","Type":"ContainerDied","Data":"d21bef21f448a972acedab3b26b8d3152f9b5e6e328f2af081ef36c3fa4d32ba"} Nov 03 20:04:00.591158 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.591139 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" Nov 03 20:04:00.591256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.591136 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71xq7q8" event={"ID":"795d21d6-9f81-44fd-a0e0-743753d41eb5","Type":"ContainerDied","Data":"58f645db1ab6cecef7b7b1f8e837b899cd27ecc2a994f957782d9afda6f3b17c"} Nov 03 20:04:00.591310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.591258 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58f645db1ab6cecef7b7b1f8e837b899cd27ecc2a994f957782d9afda6f3b17c" Nov 03 20:04:00.593219 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.593196 4457 generic.go:356] "Generic (PLEG): container finished" podID="5ed9e2be-fd53-4ae5-a3f2-c329632e79dc" containerID="1699a92378c6176e17bc7329c23f40aa0a585a39f7e59d538dc8173625053a18" exitCode=0 Nov 03 20:04:00.593302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.593267 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" event={"ID":"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc","Type":"ContainerDied","Data":"1699a92378c6176e17bc7329c23f40aa0a585a39f7e59d538dc8173625053a18"} Nov 03 20:04:00.628402 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.628316 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" podStartSLOduration=3.204846971 podStartE2EDuration="36.628297044s" totalImagesPullingTime="33.423450073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:03:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:00.626200957 +0000 UTC m=+419.001797705" watchObservedRunningTime="2026-11-03 20:04:00.628297044 +0000 UTC m=+419.003893792" Nov 03 20:04:00.711162 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.711137 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" Nov 03 20:04:00.764955 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.764924 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-util\" (UniqueName: \"kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-util\") pod \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\" (UID: \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\") " Nov 03 20:04:00.765129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.764977 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-kube-api-access-7qlnk\" (UniqueName: \"kubernetes.io/projected/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-kube-api-access-7qlnk\") pod \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\" (UID: \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\") " Nov 03 20:04:00.765129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.765027 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-bundle\") pod \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\" (UID: \"eb95a383-6a27-491e-b9e2-67c36bdeaaf0\") " Nov 03 20:04:00.765667 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.765644 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-bundle" pod "eb95a383-6a27-491e-b9e2-67c36bdeaaf0" (UID: "eb95a383-6a27-491e-b9e2-67c36bdeaaf0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:00.767098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.767079 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-kube-api-access-7qlnk" pod "eb95a383-6a27-491e-b9e2-67c36bdeaaf0" (UID: "eb95a383-6a27-491e-b9e2-67c36bdeaaf0"). InnerVolumeSpecName "kube-api-access-7qlnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:00.770488 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.770465 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-util" pod "eb95a383-6a27-491e-b9e2-67c36bdeaaf0" (UID: "eb95a383-6a27-491e-b9e2-67c36bdeaaf0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:00.865748 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.865712 4457 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-util\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:00.865748 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.865743 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qlnk\" (UniqueName: \"kubernetes.io/projected/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-kube-api-access-7qlnk\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:00.865748 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:00.865755 4457 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eb95a383-6a27-491e-b9e2-67c36bdeaaf0-bundle\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:01.599097 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.599068 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" Nov 03 20:04:01.599097 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.599072 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71qdlzr" event={"ID":"eb95a383-6a27-491e-b9e2-67c36bdeaaf0","Type":"ContainerDied","Data":"c51bef9996c080f335976a2f44040e591b81c62b8820860c3ab2ced49852ee64"} Nov 03 20:04:01.599578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.599128 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c51bef9996c080f335976a2f44040e591b81c62b8820860c3ab2ced49852ee64" Nov 03 20:04:01.736921 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.736895 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" Nov 03 20:04:01.784423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.784380 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" Nov 03 20:04:01.791781 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.791760 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" Nov 03 20:04:01.874810 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.874732 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-bundle\" (UniqueName: \"kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-bundle\") pod \"97e17443-fb36-43a0-a2d9-02b71b99d263\" (UID: \"97e17443-fb36-43a0-a2d9-02b71b99d263\") " Nov 03 20:04:01.874810 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.874777 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-util\" (UniqueName: \"kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-util\") pod \"97e17443-fb36-43a0-a2d9-02b71b99d263\" (UID: \"97e17443-fb36-43a0-a2d9-02b71b99d263\") " Nov 03 20:04:01.874810 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.874800 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-bundle\") pod \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\" (UID: \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\") " Nov 03 20:04:01.875069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.874834 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-util\" (UniqueName: \"kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-util\") pod \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\" (UID: \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\") " Nov 03 20:04:01.875069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.874853 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e17443-fb36-43a0-a2d9-02b71b99d263-kube-api-access-rrvsd\" (UniqueName: \"kubernetes.io/projected/97e17443-fb36-43a0-a2d9-02b71b99d263-kube-api-access-rrvsd\") pod \"97e17443-fb36-43a0-a2d9-02b71b99d263\" (UID: \"97e17443-fb36-43a0-a2d9-02b71b99d263\") " Nov 03 20:04:01.875069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.874894 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-util\" (UniqueName: \"kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-util\") pod \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\" (UID: \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\") " Nov 03 20:04:01.875069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.874924 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-kube-api-access-ks8p2\" (UniqueName: \"kubernetes.io/projected/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-kube-api-access-ks8p2\") pod \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\" (UID: \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\") " Nov 03 20:04:01.875257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.875063 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-kube-api-access-xjvx2\" (UniqueName: \"kubernetes.io/projected/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-kube-api-access-xjvx2\") pod \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\" (UID: \"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d\") " Nov 03 20:04:01.875257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.875131 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-bundle\") pod \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\" (UID: \"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc\") " Nov 03 20:04:01.876120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.875655 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-bundle" pod "97e17443-fb36-43a0-a2d9-02b71b99d263" (UID: "97e17443-fb36-43a0-a2d9-02b71b99d263"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:01.876120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.875947 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-bundle" pod "b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d" (UID: "b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:01.876120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.876065 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-bundle" pod "5ed9e2be-fd53-4ae5-a3f2-c329632e79dc" (UID: "5ed9e2be-fd53-4ae5-a3f2-c329632e79dc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:01.877413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.877370 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-kube-api-access-ks8p2" pod "5ed9e2be-fd53-4ae5-a3f2-c329632e79dc" (UID: "5ed9e2be-fd53-4ae5-a3f2-c329632e79dc"). InnerVolumeSpecName "kube-api-access-ks8p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:01.877571 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.877547 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e17443-fb36-43a0-a2d9-02b71b99d263-kube-api-access-rrvsd" pod "97e17443-fb36-43a0-a2d9-02b71b99d263" (UID: "97e17443-fb36-43a0-a2d9-02b71b99d263"). InnerVolumeSpecName "kube-api-access-rrvsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:01.877997 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.877969 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-kube-api-access-xjvx2" pod "b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d" (UID: "b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d"). InnerVolumeSpecName "kube-api-access-xjvx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:01.881735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.881711 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-util" pod "b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d" (UID: "b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:01.881883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.881773 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-util" pod "97e17443-fb36-43a0-a2d9-02b71b99d263" (UID: "97e17443-fb36-43a0-a2d9-02b71b99d263"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:01.882044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.882016 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-util" pod "5ed9e2be-fd53-4ae5-a3f2-c329632e79dc" (UID: "5ed9e2be-fd53-4ae5-a3f2-c329632e79dc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:01.976257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.976226 4457 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-util\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:01.976257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.976254 4457 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-bundle\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:01.976257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.976263 4457 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-util\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:01.976515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.976272 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrvsd\" (UniqueName: \"kubernetes.io/projected/97e17443-fb36-43a0-a2d9-02b71b99d263-kube-api-access-rrvsd\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:01.976515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.976281 4457 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-util\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:01.976515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.976290 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks8p2\" (UniqueName: \"kubernetes.io/projected/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-kube-api-access-ks8p2\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:01.976515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.976298 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjvx2\" (UniqueName: \"kubernetes.io/projected/b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d-kube-api-access-xjvx2\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:01.976515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.976306 4457 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ed9e2be-fd53-4ae5-a3f2-c329632e79dc-bundle\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:01.976515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:01.976314 4457 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97e17443-fb36-43a0-a2d9-02b71b99d263-bundle\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:02.604849 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:02.604819 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" Nov 03 20:04:02.605371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:02.604820 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ad6kjm" event={"ID":"b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d","Type":"ContainerDied","Data":"2bf133d2d9de96db9afa4fb47bb85c45c038a2e86a9664bb41c7978bd5646a4a"} Nov 03 20:04:02.605371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:02.604968 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bf133d2d9de96db9afa4fb47bb85c45c038a2e86a9664bb41c7978bd5646a4a" Nov 03 20:04:02.606823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:02.606802 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" Nov 03 20:04:02.606823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:02.606809 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cs9xh7" event={"ID":"5ed9e2be-fd53-4ae5-a3f2-c329632e79dc","Type":"ContainerDied","Data":"15749cffb93a77ffc2ceb9eeca07001cf7c7de1fb3fd57e77bf440aec04d4823"} Nov 03 20:04:02.606974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:02.606851 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15749cffb93a77ffc2ceb9eeca07001cf7c7de1fb3fd57e77bf440aec04d4823" Nov 03 20:04:02.608787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:02.608767 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" event={"ID":"97e17443-fb36-43a0-a2d9-02b71b99d263","Type":"ContainerDied","Data":"d4d58de634c5f5e2f7c93f988d08dd815da3928fc26e2011b55674b93cddaf10"} Nov 03 20:04:02.608876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:02.608803 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4d58de634c5f5e2f7c93f988d08dd815da3928fc26e2011b55674b93cddaf10" Nov 03 20:04:02.608876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:02.608815 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74ql6c" Nov 03 20:04:06.144319 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:04:06.144277 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:04:10.641722 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:10.641694 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:04:10.642358 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:10.642337 4457 generic.go:356] "Generic (PLEG): container finished" podID="2a47a585-8e30-4ec0-b34d-48477530d0ad" containerID="73d318a7d62039a867a27464c139f6e3acf4b83fb1e22b15d6aff78c13ebe2fb" exitCode=1 Nov 03 20:04:10.642455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:10.642382 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" event={"ID":"2a47a585-8e30-4ec0-b34d-48477530d0ad","Type":"ContainerDied","Data":"73d318a7d62039a867a27464c139f6e3acf4b83fb1e22b15d6aff78c13ebe2fb"} Nov 03 20:04:11.760152 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:11.760130 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:04:11.762019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:11.761999 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" Nov 03 20:04:11.856626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:11.856588 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a47a585-8e30-4ec0-b34d-48477530d0ad-kube-api-access-gwsbx\" (UniqueName: \"kubernetes.io/projected/2a47a585-8e30-4ec0-b34d-48477530d0ad-kube-api-access-gwsbx\") pod \"2a47a585-8e30-4ec0-b34d-48477530d0ad\" (UID: \"2a47a585-8e30-4ec0-b34d-48477530d0ad\") " Nov 03 20:04:11.856813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:11.856656 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-util\" (UniqueName: \"kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-util\") pod \"2a47a585-8e30-4ec0-b34d-48477530d0ad\" (UID: \"2a47a585-8e30-4ec0-b34d-48477530d0ad\") " Nov 03 20:04:11.856813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:11.856700 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-bundle\") pod \"2a47a585-8e30-4ec0-b34d-48477530d0ad\" (UID: \"2a47a585-8e30-4ec0-b34d-48477530d0ad\") " Nov 03 20:04:11.857409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:11.857358 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-bundle" pod "2a47a585-8e30-4ec0-b34d-48477530d0ad" (UID: "2a47a585-8e30-4ec0-b34d-48477530d0ad"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:11.858771 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:11.858749 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a47a585-8e30-4ec0-b34d-48477530d0ad-kube-api-access-gwsbx" pod "2a47a585-8e30-4ec0-b34d-48477530d0ad" (UID: "2a47a585-8e30-4ec0-b34d-48477530d0ad"). InnerVolumeSpecName "kube-api-access-gwsbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:11.862566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:11.862543 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-util" pod "2a47a585-8e30-4ec0-b34d-48477530d0ad" (UID: "2a47a585-8e30-4ec0-b34d-48477530d0ad"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:11.957587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:11.957513 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwsbx\" (UniqueName: \"kubernetes.io/projected/2a47a585-8e30-4ec0-b34d-48477530d0ad-kube-api-access-gwsbx\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:11.957587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:11.957536 4457 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-util\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:11.957587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:11.957546 4457 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2a47a585-8e30-4ec0-b34d-48477530d0ad-bundle\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:12.651552 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:12.651522 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:04:12.652203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:12.652180 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" event={"ID":"2a47a585-8e30-4ec0-b34d-48477530d0ad","Type":"ContainerDied","Data":"ab8e222f24a47ab04778debfb77e9507208a7b4ca3c48d76fd4f89345dde89dd"} Nov 03 20:04:12.652308 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:12.652223 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab8e222f24a47ab04778debfb77e9507208a7b4ca3c48d76fd4f89345dde89dd" Nov 03 20:04:12.652308 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:12.652225 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv" Nov 03 20:04:12.670958 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:04:12.670936 4457 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a47a585_8e30_4ec0_b34d_48477530d0ad.slice/crio-ab8e222f24a47ab04778debfb77e9507208a7b4ca3c48d76fd4f89345dde89dd\": RecentStats: unable to find data in memory cache]" Nov 03 20:04:13.864631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864600 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5gww6"] Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864856 4457 cpu_manager.go:414] "containerMap: removing container" podUID="eb95a383-6a27-491e-b9e2-67c36bdeaaf0" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864867 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb95a383-6a27-491e-b9e2-67c36bdeaaf0" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864882 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2a47a585-8e30-4ec0-b34d-48477530d0ad" containerName="pull" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864888 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a47a585-8e30-4ec0-b34d-48477530d0ad" containerName="pull" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864895 4457 cpu_manager.go:414] "containerMap: removing container" podUID="b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864900 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864908 4457 cpu_manager.go:414] "containerMap: removing container" podUID="eb95a383-6a27-491e-b9e2-67c36bdeaaf0" containerName="pull" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864913 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb95a383-6a27-491e-b9e2-67c36bdeaaf0" containerName="pull" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864920 4457 cpu_manager.go:414] "containerMap: removing container" podUID="5ed9e2be-fd53-4ae5-a3f2-c329632e79dc" containerName="pull" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864924 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed9e2be-fd53-4ae5-a3f2-c329632e79dc" containerName="pull" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864934 4457 cpu_manager.go:414] "containerMap: removing container" podUID="59f1e9cb-4fba-42c6-a927-f48228db3119" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864938 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f1e9cb-4fba-42c6-a927-f48228db3119" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864945 4457 cpu_manager.go:414] "containerMap: removing container" podUID="795d21d6-9f81-44fd-a0e0-743753d41eb5" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864950 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795d21d6-9f81-44fd-a0e0-743753d41eb5" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864958 4457 cpu_manager.go:414] "containerMap: removing container" podUID="795d21d6-9f81-44fd-a0e0-743753d41eb5" containerName="util" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864962 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795d21d6-9f81-44fd-a0e0-743753d41eb5" containerName="util" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864968 4457 cpu_manager.go:414] "containerMap: removing container" podUID="b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d" containerName="pull" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864973 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d" containerName="pull" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864978 4457 cpu_manager.go:414] "containerMap: removing container" podUID="bac274e7-eed4-4ccc-b597-37581ed614e7" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864982 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac274e7-eed4-4ccc-b597-37581ed614e7" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864988 4457 cpu_manager.go:414] "containerMap: removing container" podUID="b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d" containerName="util" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864992 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d" containerName="util" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.864997 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d7476a38-ebfa-48b2-a595-bf0d97d48707" containerName="util" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865002 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7476a38-ebfa-48b2-a595-bf0d97d48707" containerName="util" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865009 4457 cpu_manager.go:414] "containerMap: removing container" podUID="97e17443-fb36-43a0-a2d9-02b71b99d263" containerName="util" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865014 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e17443-fb36-43a0-a2d9-02b71b99d263" containerName="util" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865020 4457 cpu_manager.go:414] "containerMap: removing container" podUID="97e17443-fb36-43a0-a2d9-02b71b99d263" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865025 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e17443-fb36-43a0-a2d9-02b71b99d263" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865033 4457 cpu_manager.go:414] "containerMap: removing container" podUID="bac274e7-eed4-4ccc-b597-37581ed614e7" containerName="util" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865038 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac274e7-eed4-4ccc-b597-37581ed614e7" containerName="util" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865043 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2a47a585-8e30-4ec0-b34d-48477530d0ad" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865048 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a47a585-8e30-4ec0-b34d-48477530d0ad" containerName="extract" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865055 4457 cpu_manager.go:414] "containerMap: removing container" podUID="97e17443-fb36-43a0-a2d9-02b71b99d263" containerName="pull" Nov 03 20:04:13.865041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865059 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e17443-fb36-43a0-a2d9-02b71b99d263" containerName="pull" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865068 4457 cpu_manager.go:414] "containerMap: removing container" podUID="5ed9e2be-fd53-4ae5-a3f2-c329632e79dc" containerName="util" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865073 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed9e2be-fd53-4ae5-a3f2-c329632e79dc" containerName="util" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865079 4457 cpu_manager.go:414] "containerMap: removing container" podUID="795d21d6-9f81-44fd-a0e0-743753d41eb5" containerName="pull" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865083 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="795d21d6-9f81-44fd-a0e0-743753d41eb5" containerName="pull" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865088 4457 cpu_manager.go:414] "containerMap: removing container" podUID="5ed9e2be-fd53-4ae5-a3f2-c329632e79dc" containerName="extract" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865091 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed9e2be-fd53-4ae5-a3f2-c329632e79dc" containerName="extract" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865096 4457 cpu_manager.go:414] "containerMap: removing container" podUID="59f1e9cb-4fba-42c6-a927-f48228db3119" containerName="util" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865100 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f1e9cb-4fba-42c6-a927-f48228db3119" containerName="util" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865105 4457 cpu_manager.go:414] "containerMap: removing container" podUID="59f1e9cb-4fba-42c6-a927-f48228db3119" containerName="pull" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865109 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f1e9cb-4fba-42c6-a927-f48228db3119" containerName="pull" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865117 4457 cpu_manager.go:414] "containerMap: removing container" podUID="eb95a383-6a27-491e-b9e2-67c36bdeaaf0" containerName="util" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865120 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb95a383-6a27-491e-b9e2-67c36bdeaaf0" containerName="util" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865125 4457 cpu_manager.go:414] "containerMap: removing container" podUID="bac274e7-eed4-4ccc-b597-37581ed614e7" containerName="pull" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865129 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bac274e7-eed4-4ccc-b597-37581ed614e7" containerName="pull" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865134 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d7476a38-ebfa-48b2-a595-bf0d97d48707" containerName="pull" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865139 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7476a38-ebfa-48b2-a595-bf0d97d48707" containerName="pull" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865147 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d7476a38-ebfa-48b2-a595-bf0d97d48707" containerName="extract" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865152 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7476a38-ebfa-48b2-a595-bf0d97d48707" containerName="extract" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865158 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2a47a585-8e30-4ec0-b34d-48477530d0ad" containerName="util" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865162 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a47a585-8e30-4ec0-b34d-48477530d0ad" containerName="util" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865198 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7476a38-ebfa-48b2-a595-bf0d97d48707" containerName="extract" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865205 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb95a383-6a27-491e-b9e2-67c36bdeaaf0" containerName="extract" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865211 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="b669aa2f-f7ef-4d4a-92a8-fe81fc862d1d" containerName="extract" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865217 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f1e9cb-4fba-42c6-a927-f48228db3119" containerName="extract" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865221 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="795d21d6-9f81-44fd-a0e0-743753d41eb5" containerName="extract" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865226 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="bac274e7-eed4-4ccc-b597-37581ed614e7" containerName="extract" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865232 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e17443-fb36-43a0-a2d9-02b71b99d263" containerName="extract" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865237 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a47a585-8e30-4ec0-b34d-48477530d0ad" containerName="extract" Nov 03 20:04:13.866121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.865241 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed9e2be-fd53-4ae5-a3f2-c329632e79dc" containerName="extract" Nov 03 20:04:13.867274 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.867256 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gww6" Nov 03 20:04:13.878321 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.878302 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 20:04:13.878321 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.878313 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 20:04:13.878470 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.878316 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nj4dh\"" Nov 03 20:04:13.883508 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.883490 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5gww6"] Nov 03 20:04:13.973085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:13.973046 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zljm2\" (UniqueName: \"kubernetes.io/projected/a6e7ed4a-ecf6-4a3a-8037-e10cba476618-kube-api-access-zljm2\") pod \"authorino-operator-7b7bb4bc77-5gww6\" (UID: \"a6e7ed4a-ecf6-4a3a-8037-e10cba476618\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gww6" Nov 03 20:04:14.074160 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:14.074124 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljm2\" (UniqueName: \"kubernetes.io/projected/a6e7ed4a-ecf6-4a3a-8037-e10cba476618-kube-api-access-zljm2\") pod \"authorino-operator-7b7bb4bc77-5gww6\" (UID: \"a6e7ed4a-ecf6-4a3a-8037-e10cba476618\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gww6" Nov 03 20:04:14.086941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:14.086911 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zljm2\" (UniqueName: \"kubernetes.io/projected/a6e7ed4a-ecf6-4a3a-8037-e10cba476618-kube-api-access-zljm2\") pod \"authorino-operator-7b7bb4bc77-5gww6\" (UID: \"a6e7ed4a-ecf6-4a3a-8037-e10cba476618\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gww6" Nov 03 20:04:14.178324 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:14.178238 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gww6" Nov 03 20:04:14.452710 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:14.452686 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-5gww6"] Nov 03 20:04:14.454924 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:04:14.454896 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e7ed4a_ecf6_4a3a_8037_e10cba476618.slice/crio-16f6e199182a471f31ab736e825d42e36d438667aac649a382d60d7b2dd31fb1 WatchSource:0}: Error finding container 16f6e199182a471f31ab736e825d42e36d438667aac649a382d60d7b2dd31fb1: Status 404 returned error can't find the container with id 16f6e199182a471f31ab736e825d42e36d438667aac649a382d60d7b2dd31fb1 Nov 03 20:04:14.665044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:14.665008 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gww6" event={"ID":"a6e7ed4a-ecf6-4a3a-8037-e10cba476618","Type":"ContainerStarted","Data":"16f6e199182a471f31ab736e825d42e36d438667aac649a382d60d7b2dd31fb1"} Nov 03 20:04:17.576474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:17.576444 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-jth7z"] Nov 03 20:04:17.578615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:17.578596 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jth7z" Nov 03 20:04:17.584860 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:17.584841 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5g689\"" Nov 03 20:04:17.594742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:17.594715 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-jth7z"] Nov 03 20:04:17.683611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:17.683572 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gww6" event={"ID":"a6e7ed4a-ecf6-4a3a-8037-e10cba476618","Type":"ContainerStarted","Data":"936e1bfee728b12cd1e5518d6e160f0c13dff1b0f0d8ee8d4e14f9a49baa83b0"} Nov 03 20:04:17.683778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:17.683637 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gww6" Nov 03 20:04:17.705514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:17.705471 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gww6" podStartSLOduration=2.051031317 podStartE2EDuration="4.705459533s" totalImagesPullingTime="2.654428216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:17.705211416 +0000 UTC m=+436.080808168" watchObservedRunningTime="2026-11-03 20:04:17.705459533 +0000 UTC m=+436.081056281" Nov 03 20:04:17.706180 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:17.706151 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbwbk\" (UniqueName: \"kubernetes.io/projected/603fa1ee-3000-485f-92ad-66a3f1bd726c-kube-api-access-vbwbk\") pod \"limitador-operator-controller-manager-5778bb4449-jth7z\" (UID: \"603fa1ee-3000-485f-92ad-66a3f1bd726c\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jth7z" Nov 03 20:04:17.807474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:17.807438 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwbk\" (UniqueName: \"kubernetes.io/projected/603fa1ee-3000-485f-92ad-66a3f1bd726c-kube-api-access-vbwbk\") pod \"limitador-operator-controller-manager-5778bb4449-jth7z\" (UID: \"603fa1ee-3000-485f-92ad-66a3f1bd726c\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jth7z" Nov 03 20:04:17.820423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:17.820364 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbwbk\" (UniqueName: \"kubernetes.io/projected/603fa1ee-3000-485f-92ad-66a3f1bd726c-kube-api-access-vbwbk\") pod \"limitador-operator-controller-manager-5778bb4449-jth7z\" (UID: \"603fa1ee-3000-485f-92ad-66a3f1bd726c\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jth7z" Nov 03 20:04:17.890572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:17.890488 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jth7z" Nov 03 20:04:18.074438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.074408 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd"] Nov 03 20:04:18.078347 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.078317 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.090163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.090140 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 20:04:18.090528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.090507 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 20:04:18.090588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.090529 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 20:04:18.091130 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.091116 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 20:04:18.091164 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.091129 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-t7ddc\"" Nov 03 20:04:18.091431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.091414 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 20:04:18.179031 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.179006 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-jth7z"] Nov 03 20:04:18.179954 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:04:18.179928 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod603fa1ee_3000_485f_92ad_66a3f1bd726c.slice/crio-c65053571bc5dbe1cae342022010c8819ce1caceb04ba52e4940e68b41529d5c WatchSource:0}: Error finding container c65053571bc5dbe1cae342022010c8819ce1caceb04ba52e4940e68b41529d5c: Status 404 returned error can't find the container with id c65053571bc5dbe1cae342022010c8819ce1caceb04ba52e4940e68b41529d5c Nov 03 20:04:18.210867 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.210836 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81c8482e-e941-49ed-ad35-73866f7e6bed-manifests\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.211030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.210871 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81c8482e-e941-49ed-ad35-73866f7e6bed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.211030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.211003 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81c8482e-e941-49ed-ad35-73866f7e6bed-webhook-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.211177 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.211042 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndxv4\" (UniqueName: \"kubernetes.io/projected/81c8482e-e941-49ed-ad35-73866f7e6bed-kube-api-access-ndxv4\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.211177 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.211076 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81c8482e-e941-49ed-ad35-73866f7e6bed-metrics-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.211261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.211201 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81c8482e-e941-49ed-ad35-73866f7e6bed-tmp\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.213511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.213486 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd"] Nov 03 20:04:18.312430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.312373 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81c8482e-e941-49ed-ad35-73866f7e6bed-tmp\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.312596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.312457 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81c8482e-e941-49ed-ad35-73866f7e6bed-manifests\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.312596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.312480 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81c8482e-e941-49ed-ad35-73866f7e6bed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.312596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.312496 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81c8482e-e941-49ed-ad35-73866f7e6bed-webhook-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.312596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.312511 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndxv4\" (UniqueName: \"kubernetes.io/projected/81c8482e-e941-49ed-ad35-73866f7e6bed-kube-api-access-ndxv4\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.312596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.312532 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81c8482e-e941-49ed-ad35-73866f7e6bed-metrics-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.312917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.312896 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/81c8482e-e941-49ed-ad35-73866f7e6bed-manifests\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.315013 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.314984 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81c8482e-e941-49ed-ad35-73866f7e6bed-tmp\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.315216 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.315191 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/81c8482e-e941-49ed-ad35-73866f7e6bed-metrics-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.315276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.315191 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81c8482e-e941-49ed-ad35-73866f7e6bed-webhook-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.315276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.315202 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/81c8482e-e941-49ed-ad35-73866f7e6bed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.324873 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.324848 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndxv4\" (UniqueName: \"kubernetes.io/projected/81c8482e-e941-49ed-ad35-73866f7e6bed-kube-api-access-ndxv4\") pod \"opendatahub-operator-controller-manager-79746d584b-zn6wd\" (UID: \"81c8482e-e941-49ed-ad35-73866f7e6bed\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.391941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.391904 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:18.689738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.689707 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jth7z" event={"ID":"603fa1ee-3000-485f-92ad-66a3f1bd726c","Type":"ContainerStarted","Data":"c65053571bc5dbe1cae342022010c8819ce1caceb04ba52e4940e68b41529d5c"} Nov 03 20:04:18.692223 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:04:18.692200 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c8482e_e941_49ed_ad35_73866f7e6bed.slice/crio-298f65e9266f40bc9059cc0c5f5519feea2bdd8b04bf55677804f7e3f5f8dbdc WatchSource:0}: Error finding container 298f65e9266f40bc9059cc0c5f5519feea2bdd8b04bf55677804f7e3f5f8dbdc: Status 404 returned error can't find the container with id 298f65e9266f40bc9059cc0c5f5519feea2bdd8b04bf55677804f7e3f5f8dbdc Nov 03 20:04:18.698718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:18.698697 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd"] Nov 03 20:04:19.029324 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.029246 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54"] Nov 03 20:04:19.032426 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.032407 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" Nov 03 20:04:19.036148 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.036129 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 20:04:19.036494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.036475 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-ft92x\"" Nov 03 20:04:19.039149 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.039131 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 20:04:19.044834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.044813 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54"] Nov 03 20:04:19.119955 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.119915 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54\" (UID: \"119f17d8-60da-4c98-92f4-8fc45ee471a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" Nov 03 20:04:19.120129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.120058 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hsxb\" (UniqueName: \"kubernetes.io/projected/119f17d8-60da-4c98-92f4-8fc45ee471a3-kube-api-access-8hsxb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54\" (UID: \"119f17d8-60da-4c98-92f4-8fc45ee471a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" Nov 03 20:04:19.120129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.120111 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54\" (UID: \"119f17d8-60da-4c98-92f4-8fc45ee471a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" Nov 03 20:04:19.220854 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.220816 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hsxb\" (UniqueName: \"kubernetes.io/projected/119f17d8-60da-4c98-92f4-8fc45ee471a3-kube-api-access-8hsxb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54\" (UID: \"119f17d8-60da-4c98-92f4-8fc45ee471a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" Nov 03 20:04:19.221028 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.220885 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54\" (UID: \"119f17d8-60da-4c98-92f4-8fc45ee471a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" Nov 03 20:04:19.221028 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.220938 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54\" (UID: \"119f17d8-60da-4c98-92f4-8fc45ee471a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" Nov 03 20:04:19.221347 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.221320 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54\" (UID: \"119f17d8-60da-4c98-92f4-8fc45ee471a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" Nov 03 20:04:19.221443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.221421 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54\" (UID: \"119f17d8-60da-4c98-92f4-8fc45ee471a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" Nov 03 20:04:19.242246 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.242200 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hsxb\" (UniqueName: \"kubernetes.io/projected/119f17d8-60da-4c98-92f4-8fc45ee471a3-kube-api-access-8hsxb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54\" (UID: \"119f17d8-60da-4c98-92f4-8fc45ee471a3\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" Nov 03 20:04:19.345567 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.345300 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" Nov 03 20:04:19.642977 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.642942 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54"] Nov 03 20:04:19.646084 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:04:19.646055 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod119f17d8_60da_4c98_92f4_8fc45ee471a3.slice/crio-27c20a9ed6ef3e32f3cbaf216fce9ca819f9541f0217cf4550ded5097c53ba72 WatchSource:0}: Error finding container 27c20a9ed6ef3e32f3cbaf216fce9ca819f9541f0217cf4550ded5097c53ba72: Status 404 returned error can't find the container with id 27c20a9ed6ef3e32f3cbaf216fce9ca819f9541f0217cf4550ded5097c53ba72 Nov 03 20:04:19.698439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.698358 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" event={"ID":"81c8482e-e941-49ed-ad35-73866f7e6bed","Type":"ContainerStarted","Data":"298f65e9266f40bc9059cc0c5f5519feea2bdd8b04bf55677804f7e3f5f8dbdc"} Nov 03 20:04:19.699764 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:19.699740 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" event={"ID":"119f17d8-60da-4c98-92f4-8fc45ee471a3","Type":"ContainerStarted","Data":"27c20a9ed6ef3e32f3cbaf216fce9ca819f9541f0217cf4550ded5097c53ba72"} Nov 03 20:04:20.706428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:20.706374 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jth7z" event={"ID":"603fa1ee-3000-485f-92ad-66a3f1bd726c","Type":"ContainerStarted","Data":"dd67069925b2e7223c7447339f9b1e3508e14a1518e1c6f26941dafcb7d8bcc5"} Nov 03 20:04:20.706894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:20.706456 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jth7z" Nov 03 20:04:20.708020 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:20.707994 4457 generic.go:356] "Generic (PLEG): container finished" podID="119f17d8-60da-4c98-92f4-8fc45ee471a3" containerID="6cf54e30ccd42026d15c70145e915c3e76dc99daba681565240d54478b80cc63" exitCode=0 Nov 03 20:04:20.708168 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:20.708067 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" event={"ID":"119f17d8-60da-4c98-92f4-8fc45ee471a3","Type":"ContainerDied","Data":"6cf54e30ccd42026d15c70145e915c3e76dc99daba681565240d54478b80cc63"} Nov 03 20:04:20.731270 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:20.729524 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jth7z" podStartSLOduration=1.842056892 podStartE2EDuration="3.72950991s" totalImagesPullingTime="1.887453018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:20.728014608 +0000 UTC m=+439.103611369" watchObservedRunningTime="2026-11-03 20:04:20.72950991 +0000 UTC m=+439.105106657" Nov 03 20:04:21.718546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:21.718513 4457 generic.go:356] "Generic (PLEG): container finished" podID="119f17d8-60da-4c98-92f4-8fc45ee471a3" containerID="6fd9f8c2200d0b72987af362f8dc6fb03ed6c1ab1405e2203762a862db213513" exitCode=0 Nov 03 20:04:21.719031 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:21.718597 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" event={"ID":"119f17d8-60da-4c98-92f4-8fc45ee471a3","Type":"ContainerDied","Data":"6fd9f8c2200d0b72987af362f8dc6fb03ed6c1ab1405e2203762a862db213513"} Nov 03 20:04:22.725300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:22.725265 4457 generic.go:356] "Generic (PLEG): container finished" podID="119f17d8-60da-4c98-92f4-8fc45ee471a3" containerID="70ee2bc4db9b415e7f47f1e393daf6cf43fe2fc5af6bda0995b201f1f36d5e7d" exitCode=0 Nov 03 20:04:22.725759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:22.725346 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" event={"ID":"119f17d8-60da-4c98-92f4-8fc45ee471a3","Type":"ContainerDied","Data":"70ee2bc4db9b415e7f47f1e393daf6cf43fe2fc5af6bda0995b201f1f36d5e7d"} Nov 03 20:04:22.727030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:22.727004 4457 generic.go:356] "Generic (PLEG): container finished" podID="81c8482e-e941-49ed-ad35-73866f7e6bed" containerID="83738913b9f44a54eecf339070d0cf2d21a5d0fd01711d3bbb7291ae036f556b" exitCode=0 Nov 03 20:04:22.727133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:22.727086 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" event={"ID":"81c8482e-e941-49ed-ad35-73866f7e6bed","Type":"ContainerDied","Data":"83738913b9f44a54eecf339070d0cf2d21a5d0fd01711d3bbb7291ae036f556b"} Nov 03 20:04:23.733280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:23.733243 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" event={"ID":"81c8482e-e941-49ed-ad35-73866f7e6bed","Type":"ContainerStarted","Data":"70e28939881575136760cd12f758d44f018c72a2ca81af30906aaaa1a1c75b6a"} Nov 03 20:04:23.778971 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:23.778921 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" podStartSLOduration=1.531251149 podStartE2EDuration="5.778900679s" totalImagesPullingTime="3.784753083s" totalInitContainerRuntime="462.896447ms" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:23.777992899 +0000 UTC m=+442.153589646" watchObservedRunningTime="2026-11-03 20:04:23.778900679 +0000 UTC m=+442.154497427" Nov 03 20:04:23.858348 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:23.858325 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" Nov 03 20:04:23.959178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:23.959139 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-util\" (UniqueName: \"kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-util\") pod \"119f17d8-60da-4c98-92f4-8fc45ee471a3\" (UID: \"119f17d8-60da-4c98-92f4-8fc45ee471a3\") " Nov 03 20:04:23.959178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:23.959177 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-bundle\") pod \"119f17d8-60da-4c98-92f4-8fc45ee471a3\" (UID: \"119f17d8-60da-4c98-92f4-8fc45ee471a3\") " Nov 03 20:04:23.959455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:23.959212 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/119f17d8-60da-4c98-92f4-8fc45ee471a3-kube-api-access-8hsxb\" (UniqueName: \"kubernetes.io/projected/119f17d8-60da-4c98-92f4-8fc45ee471a3-kube-api-access-8hsxb\") pod \"119f17d8-60da-4c98-92f4-8fc45ee471a3\" (UID: \"119f17d8-60da-4c98-92f4-8fc45ee471a3\") " Nov 03 20:04:23.959941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:23.959910 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-bundle" pod "119f17d8-60da-4c98-92f4-8fc45ee471a3" (UID: "119f17d8-60da-4c98-92f4-8fc45ee471a3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:23.961351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:23.961322 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/119f17d8-60da-4c98-92f4-8fc45ee471a3-kube-api-access-8hsxb" pod "119f17d8-60da-4c98-92f4-8fc45ee471a3" (UID: "119f17d8-60da-4c98-92f4-8fc45ee471a3"). InnerVolumeSpecName "kube-api-access-8hsxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:04:23.965526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:23.965501 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-util" pod "119f17d8-60da-4c98-92f4-8fc45ee471a3" (UID: "119f17d8-60da-4c98-92f4-8fc45ee471a3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:04:24.060537 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:24.060510 4457 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-util\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:24.060537 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:24.060537 4457 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/119f17d8-60da-4c98-92f4-8fc45ee471a3-bundle\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:24.060720 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:24.060548 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hsxb\" (UniqueName: \"kubernetes.io/projected/119f17d8-60da-4c98-92f4-8fc45ee471a3-kube-api-access-8hsxb\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:04:24.744525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:24.744492 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" Nov 03 20:04:24.745030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:24.744522 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chqf54" event={"ID":"119f17d8-60da-4c98-92f4-8fc45ee471a3","Type":"ContainerDied","Data":"27c20a9ed6ef3e32f3cbaf216fce9ca819f9541f0217cf4550ded5097c53ba72"} Nov 03 20:04:24.745030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:24.744590 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27c20a9ed6ef3e32f3cbaf216fce9ca819f9541f0217cf4550ded5097c53ba72" Nov 03 20:04:24.745030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:24.744894 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:25.855049 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:25.855007 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g"] Nov 03 20:04:25.855542 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:25.855476 4457 cpu_manager.go:414] "containerMap: removing container" podUID="119f17d8-60da-4c98-92f4-8fc45ee471a3" containerName="pull" Nov 03 20:04:25.855542 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:25.855494 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119f17d8-60da-4c98-92f4-8fc45ee471a3" containerName="pull" Nov 03 20:04:25.855542 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:25.855519 4457 cpu_manager.go:414] "containerMap: removing container" podUID="119f17d8-60da-4c98-92f4-8fc45ee471a3" containerName="util" Nov 03 20:04:25.855542 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:25.855526 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119f17d8-60da-4c98-92f4-8fc45ee471a3" containerName="util" Nov 03 20:04:25.855542 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:25.855537 4457 cpu_manager.go:414] "containerMap: removing container" podUID="119f17d8-60da-4c98-92f4-8fc45ee471a3" containerName="extract" Nov 03 20:04:25.855542 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:25.855544 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="119f17d8-60da-4c98-92f4-8fc45ee471a3" containerName="extract" Nov 03 20:04:25.855870 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:25.855637 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="119f17d8-60da-4c98-92f4-8fc45ee471a3" containerName="extract" Nov 03 20:04:25.859055 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:25.859036 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" Nov 03 20:04:25.867228 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:25.867209 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kkz74\"" Nov 03 20:04:25.873108 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:25.873088 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g"] Nov 03 20:04:25.977092 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:25.977059 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtcvc\" (UniqueName: \"kubernetes.io/projected/ee01a4bb-5bbb-4824-820f-12a621202e80-kube-api-access-xtcvc\") pod \"kuadrant-operator-controller-manager-678d87766f-8ls6g\" (UID: \"ee01a4bb-5bbb-4824-820f-12a621202e80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" Nov 03 20:04:25.977261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:25.977095 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee01a4bb-5bbb-4824-820f-12a621202e80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8ls6g\" (UID: \"ee01a4bb-5bbb-4824-820f-12a621202e80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" Nov 03 20:04:26.077687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:26.077651 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtcvc\" (UniqueName: \"kubernetes.io/projected/ee01a4bb-5bbb-4824-820f-12a621202e80-kube-api-access-xtcvc\") pod \"kuadrant-operator-controller-manager-678d87766f-8ls6g\" (UID: \"ee01a4bb-5bbb-4824-820f-12a621202e80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" Nov 03 20:04:26.077687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:26.077688 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee01a4bb-5bbb-4824-820f-12a621202e80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8ls6g\" (UID: \"ee01a4bb-5bbb-4824-820f-12a621202e80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" Nov 03 20:04:26.078051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:26.078035 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee01a4bb-5bbb-4824-820f-12a621202e80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-8ls6g\" (UID: \"ee01a4bb-5bbb-4824-820f-12a621202e80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" Nov 03 20:04:26.093466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:26.093434 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtcvc\" (UniqueName: \"kubernetes.io/projected/ee01a4bb-5bbb-4824-820f-12a621202e80-kube-api-access-xtcvc\") pod \"kuadrant-operator-controller-manager-678d87766f-8ls6g\" (UID: \"ee01a4bb-5bbb-4824-820f-12a621202e80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" Nov 03 20:04:26.171425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:26.171336 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" Nov 03 20:04:26.492146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:26.492115 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g"] Nov 03 20:04:26.492540 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:04:26.492512 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee01a4bb_5bbb_4824_820f_12a621202e80.slice/crio-737b65778678b2888355ca4211a2d0efb5709c5bc4b2ed1da15a09bd196c02cc WatchSource:0}: Error finding container 737b65778678b2888355ca4211a2d0efb5709c5bc4b2ed1da15a09bd196c02cc: Status 404 returned error can't find the container with id 737b65778678b2888355ca4211a2d0efb5709c5bc4b2ed1da15a09bd196c02cc Nov 03 20:04:26.754829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:26.754745 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" event={"ID":"ee01a4bb-5bbb-4824-820f-12a621202e80","Type":"ContainerStarted","Data":"737b65778678b2888355ca4211a2d0efb5709c5bc4b2ed1da15a09bd196c02cc"} Nov 03 20:04:28.691897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:28.691868 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-5gww6" Nov 03 20:04:31.720861 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:31.720829 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jth7z" Nov 03 20:04:32.784372 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:32.784334 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" event={"ID":"ee01a4bb-5bbb-4824-820f-12a621202e80","Type":"ContainerStarted","Data":"27c29d6e9d9ed771d1d76c037733b84c81ba674fcb18e36a1a205a126a466dc4"} Nov 03 20:04:32.784817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:32.784432 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" Nov 03 20:04:32.810171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:32.810116 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" podStartSLOduration=2.187462776 podStartE2EDuration="7.810101789s" totalImagesPullingTime="5.622639013s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:32.807064088 +0000 UTC m=+451.182660835" watchObservedRunningTime="2026-11-03 20:04:32.810101789 +0000 UTC m=+451.185698537" Nov 03 20:04:35.754302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:35.754272 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-79746d584b-zn6wd" Nov 03 20:04:35.923966 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:35.923929 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs"] Nov 03 20:04:35.928590 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:35.928570 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs" Nov 03 20:04:35.932631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:35.932611 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 20:04:35.932631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:35.932615 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7cxv2\"" Nov 03 20:04:35.932764 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:35.932630 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 20:04:35.941608 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:35.941587 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs"] Nov 03 20:04:35.963797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:35.963761 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xb8\" (UniqueName: \"kubernetes.io/projected/d8b44a4d-0b64-4984-a25a-2fd6173010b2-kube-api-access-p6xb8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vckqs\" (UID: \"d8b44a4d-0b64-4984-a25a-2fd6173010b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs" Nov 03 20:04:35.963935 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:35.963809 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8b44a4d-0b64-4984-a25a-2fd6173010b2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vckqs\" (UID: \"d8b44a4d-0b64-4984-a25a-2fd6173010b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs" Nov 03 20:04:36.065111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:36.065077 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8b44a4d-0b64-4984-a25a-2fd6173010b2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vckqs\" (UID: \"d8b44a4d-0b64-4984-a25a-2fd6173010b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs" Nov 03 20:04:36.065293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:36.065162 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xb8\" (UniqueName: \"kubernetes.io/projected/d8b44a4d-0b64-4984-a25a-2fd6173010b2-kube-api-access-p6xb8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vckqs\" (UID: \"d8b44a4d-0b64-4984-a25a-2fd6173010b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs" Nov 03 20:04:36.065524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:36.065501 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8b44a4d-0b64-4984-a25a-2fd6173010b2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vckqs\" (UID: \"d8b44a4d-0b64-4984-a25a-2fd6173010b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs" Nov 03 20:04:36.084427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:36.084405 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xb8\" (UniqueName: \"kubernetes.io/projected/d8b44a4d-0b64-4984-a25a-2fd6173010b2-kube-api-access-p6xb8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-vckqs\" (UID: \"d8b44a4d-0b64-4984-a25a-2fd6173010b2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs" Nov 03 20:04:36.143656 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:04:36.143620 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:04:36.239574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:36.239542 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs" Nov 03 20:04:36.573651 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:36.573627 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs"] Nov 03 20:04:36.576675 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:04:36.576652 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b44a4d_0b64_4984_a25a_2fd6173010b2.slice/crio-aa4513012b117080b8fcc7bd1abbe97ad03d914fac22e6c60d0408225dd0f2c4 WatchSource:0}: Error finding container aa4513012b117080b8fcc7bd1abbe97ad03d914fac22e6c60d0408225dd0f2c4: Status 404 returned error can't find the container with id aa4513012b117080b8fcc7bd1abbe97ad03d914fac22e6c60d0408225dd0f2c4 Nov 03 20:04:36.802864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:36.802830 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs" event={"ID":"d8b44a4d-0b64-4984-a25a-2fd6173010b2","Type":"ContainerStarted","Data":"aa4513012b117080b8fcc7bd1abbe97ad03d914fac22e6c60d0408225dd0f2c4"} Nov 03 20:04:38.815003 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:38.814962 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs" event={"ID":"d8b44a4d-0b64-4984-a25a-2fd6173010b2","Type":"ContainerStarted","Data":"c96b9275205ca207f4a18c9d659ac18dbde68d40cacc6a644fd42998b2817e83"} Nov 03 20:04:38.856005 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:38.855955 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-vckqs" podStartSLOduration=2.147273922 podStartE2EDuration="3.855941215s" totalImagesPullingTime="1.708667293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:04:38.854218963 +0000 UTC m=+457.229815710" watchObservedRunningTime="2026-11-03 20:04:38.855941215 +0000 UTC m=+457.231537962" Nov 03 20:04:43.791435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:04:43.791381 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" Nov 03 20:05:01.159173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.159134 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6"] Nov 03 20:05:01.167877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.167849 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.177710 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.177642 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-8bd2m\"" Nov 03 20:05:01.178835 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.178775 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6"] Nov 03 20:05:01.279801 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.279757 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.279801 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.279803 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.280017 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.279891 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.280017 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.279925 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.280017 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.279964 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm9zx\" (UniqueName: \"kubernetes.io/projected/82519fb3-ab4a-42bf-a8e0-c911224c33ab-kube-api-access-jm9zx\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.280126 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.280079 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.280199 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.280183 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.280262 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.280217 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.280314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.280260 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.380866 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.380819 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm9zx\" (UniqueName: \"kubernetes.io/projected/82519fb3-ab4a-42bf-a8e0-c911224c33ab-kube-api-access-jm9zx\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.381043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.380880 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.381043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.380945 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.381043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.380973 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.381043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.381011 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.381261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.381049 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.381261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.381075 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.381261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.381118 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.381261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.381151 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.381493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.381348 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.381493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.381471 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.381592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.381534 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istio-data\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.381954 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.381928 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.382198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.382158 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.389416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.384279 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.389416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.386877 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.396648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.396623 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm9zx\" (UniqueName: \"kubernetes.io/projected/82519fb3-ab4a-42bf-a8e0-c911224c33ab-kube-api-access-jm9zx\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.396984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.396965 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82519fb3-ab4a-42bf-a8e0-c911224c33ab-istio-token\") pod \"data-science-gateway-data-science-gateway-class-57dc67465-qrjl6\" (UID: \"82519fb3-ab4a-42bf-a8e0-c911224c33ab\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.483480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.483379 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:01.709350 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.709317 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xg8rg"] Nov 03 20:05:01.714708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.714195 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" Nov 03 20:05:01.727687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.727235 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 03 20:05:01.728351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.728321 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qcqxj\"" Nov 03 20:05:01.730975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.730953 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xg8rg"] Nov 03 20:05:01.784326 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.784294 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2s9r\" (UniqueName: \"kubernetes.io/projected/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-kube-api-access-w2s9r\") pod \"limitador-limitador-74dfcfb894-xg8rg\" (UID: \"fa3f12f8-4195-46f0-8e87-f1da7e8400f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" Nov 03 20:05:01.784485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.784463 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-config-file\") pod \"limitador-limitador-74dfcfb894-xg8rg\" (UID: \"fa3f12f8-4195-46f0-8e87-f1da7e8400f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" Nov 03 20:05:01.785918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.785882 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:05:01.786013 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.785959 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:05:01.786013 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.785998 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:05:01.786013 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.786004 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6"] Nov 03 20:05:01.810595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.810565 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xg8rg"] Nov 03 20:05:01.885280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.885247 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-config-file\") pod \"limitador-limitador-74dfcfb894-xg8rg\" (UID: \"fa3f12f8-4195-46f0-8e87-f1da7e8400f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" Nov 03 20:05:01.885502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.885331 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2s9r\" (UniqueName: \"kubernetes.io/projected/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-kube-api-access-w2s9r\") pod \"limitador-limitador-74dfcfb894-xg8rg\" (UID: \"fa3f12f8-4195-46f0-8e87-f1da7e8400f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" Nov 03 20:05:01.886102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.886077 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-config-file\") pod \"limitador-limitador-74dfcfb894-xg8rg\" (UID: \"fa3f12f8-4195-46f0-8e87-f1da7e8400f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" Nov 03 20:05:01.898020 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.897990 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2s9r\" (UniqueName: \"kubernetes.io/projected/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-kube-api-access-w2s9r\") pod \"limitador-limitador-74dfcfb894-xg8rg\" (UID: \"fa3f12f8-4195-46f0-8e87-f1da7e8400f7\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" Nov 03 20:05:01.931665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.931628 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" event={"ID":"82519fb3-ab4a-42bf-a8e0-c911224c33ab","Type":"ContainerStarted","Data":"9442d73ddfb417af4f9e191ff3ea66558049cd6b3d31e8c12e0ef3c7b8c7c9a5"} Nov 03 20:05:01.931945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:01.931922 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" event={"ID":"82519fb3-ab4a-42bf-a8e0-c911224c33ab","Type":"ContainerStarted","Data":"3a1a1e30587ea25551dd91fde850fd632c0470101abf268702ab2ea70ece5ecd"} Nov 03 20:05:02.045345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.045305 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" Nov 03 20:05:02.342768 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.342695 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" podStartSLOduration=1.342673962 podStartE2EDuration="1.342673962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:01.957893749 +0000 UTC m=+480.333490497" watchObservedRunningTime="2026-11-03 20:05:02.342673962 +0000 UTC m=+480.718270711" Nov 03 20:05:02.343193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.343033 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xg8rg"] Nov 03 20:05:02.344878 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:05:02.344839 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa3f12f8_4195_46f0_8e87_f1da7e8400f7.slice/crio-5181ebbe33de23acc749bc86e0a13d489f2affdabf84d8f36f098628929cb0b5 WatchSource:0}: Error finding container 5181ebbe33de23acc749bc86e0a13d489f2affdabf84d8f36f098628929cb0b5: Status 404 returned error can't find the container with id 5181ebbe33de23acc749bc86e0a13d489f2affdabf84d8f36f098628929cb0b5 Nov 03 20:05:02.483703 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.483569 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:02.488641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.488615 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:02.869577 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.869540 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-djrjk"] Nov 03 20:05:02.909874 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.909831 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-djrjk"] Nov 03 20:05:02.910051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.909976 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-djrjk" Nov 03 20:05:02.919535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.919485 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-grh7n\"" Nov 03 20:05:02.938618 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.938586 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" event={"ID":"fa3f12f8-4195-46f0-8e87-f1da7e8400f7","Type":"ContainerStarted","Data":"5181ebbe33de23acc749bc86e0a13d489f2affdabf84d8f36f098628929cb0b5"} Nov 03 20:05:02.955453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.955420 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:02.956307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.956284 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-57dc67465-qrjl6" Nov 03 20:05:02.959059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.959037 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5jhck"] Nov 03 20:05:02.978366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.978339 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5jhck"] Nov 03 20:05:02.978513 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.978444 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5jhck" Nov 03 20:05:02.996260 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:02.996234 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4rfl\" (UniqueName: \"kubernetes.io/projected/c4bdbb41-d890-4509-b6c2-3e126f4f45a0-kube-api-access-n4rfl\") pod \"authorino-57f7f8659f-djrjk\" (UID: \"c4bdbb41-d890-4509-b6c2-3e126f4f45a0\") " pod="kuadrant-system/authorino-57f7f8659f-djrjk" Nov 03 20:05:03.097664 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:03.097592 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4rfl\" (UniqueName: \"kubernetes.io/projected/c4bdbb41-d890-4509-b6c2-3e126f4f45a0-kube-api-access-n4rfl\") pod \"authorino-57f7f8659f-djrjk\" (UID: \"c4bdbb41-d890-4509-b6c2-3e126f4f45a0\") " pod="kuadrant-system/authorino-57f7f8659f-djrjk" Nov 03 20:05:03.097847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:03.097702 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chdfh\" (UniqueName: \"kubernetes.io/projected/26273dd2-22e7-419d-bbc0-cf1f831ef5f8-kube-api-access-chdfh\") pod \"authorino-56c6c9ddd5-5jhck\" (UID: \"26273dd2-22e7-419d-bbc0-cf1f831ef5f8\") " pod="kuadrant-system/authorino-56c6c9ddd5-5jhck" Nov 03 20:05:03.114611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:03.114576 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4rfl\" (UniqueName: \"kubernetes.io/projected/c4bdbb41-d890-4509-b6c2-3e126f4f45a0-kube-api-access-n4rfl\") pod \"authorino-57f7f8659f-djrjk\" (UID: \"c4bdbb41-d890-4509-b6c2-3e126f4f45a0\") " pod="kuadrant-system/authorino-57f7f8659f-djrjk" Nov 03 20:05:03.199043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:03.198955 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chdfh\" (UniqueName: \"kubernetes.io/projected/26273dd2-22e7-419d-bbc0-cf1f831ef5f8-kube-api-access-chdfh\") pod \"authorino-56c6c9ddd5-5jhck\" (UID: \"26273dd2-22e7-419d-bbc0-cf1f831ef5f8\") " pod="kuadrant-system/authorino-56c6c9ddd5-5jhck" Nov 03 20:05:03.211241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:03.211214 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chdfh\" (UniqueName: \"kubernetes.io/projected/26273dd2-22e7-419d-bbc0-cf1f831ef5f8-kube-api-access-chdfh\") pod \"authorino-56c6c9ddd5-5jhck\" (UID: \"26273dd2-22e7-419d-bbc0-cf1f831ef5f8\") " pod="kuadrant-system/authorino-56c6c9ddd5-5jhck" Nov 03 20:05:03.256120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:03.256092 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-djrjk" Nov 03 20:05:03.297253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:03.297216 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5jhck" Nov 03 20:05:03.538519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:03.538490 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-djrjk"] Nov 03 20:05:03.541019 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:05:03.540990 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4bdbb41_d890_4509_b6c2_3e126f4f45a0.slice/crio-7887d627d5f0e73fc433f9f0acd8d24e44f19175fa53b8962780d96242dc2f22 WatchSource:0}: Error finding container 7887d627d5f0e73fc433f9f0acd8d24e44f19175fa53b8962780d96242dc2f22: Status 404 returned error can't find the container with id 7887d627d5f0e73fc433f9f0acd8d24e44f19175fa53b8962780d96242dc2f22 Nov 03 20:05:03.578459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:03.578437 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5jhck"] Nov 03 20:05:03.580798 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:05:03.580770 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26273dd2_22e7_419d_bbc0_cf1f831ef5f8.slice/crio-06c66bb9e3bf55588c6a77233d26b51b284e596aec74d3b0e8d3a9592a7c35b5 WatchSource:0}: Error finding container 06c66bb9e3bf55588c6a77233d26b51b284e596aec74d3b0e8d3a9592a7c35b5: Status 404 returned error can't find the container with id 06c66bb9e3bf55588c6a77233d26b51b284e596aec74d3b0e8d3a9592a7c35b5 Nov 03 20:05:03.944830 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:03.944791 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5jhck" event={"ID":"26273dd2-22e7-419d-bbc0-cf1f831ef5f8","Type":"ContainerStarted","Data":"06c66bb9e3bf55588c6a77233d26b51b284e596aec74d3b0e8d3a9592a7c35b5"} Nov 03 20:05:03.945935 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:03.945907 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-djrjk" event={"ID":"c4bdbb41-d890-4509-b6c2-3e126f4f45a0","Type":"ContainerStarted","Data":"7887d627d5f0e73fc433f9f0acd8d24e44f19175fa53b8962780d96242dc2f22"} Nov 03 20:05:05.958239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:05.958143 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5jhck" event={"ID":"26273dd2-22e7-419d-bbc0-cf1f831ef5f8","Type":"ContainerStarted","Data":"6a84c99cd862fccb71661bacec9899ecaeab348650d3dc606f7ed4c35866176c"} Nov 03 20:05:05.959830 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:05.959802 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-djrjk" event={"ID":"c4bdbb41-d890-4509-b6c2-3e126f4f45a0","Type":"ContainerStarted","Data":"4d3188a57ac587d8c4107bf3f5ef8d4ae722d014bf093a0bf09e98bf53f5da52"} Nov 03 20:05:05.976380 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:05.976332 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-5jhck" podStartSLOduration=1.9818990969999999 podStartE2EDuration="3.976318242s" totalImagesPullingTime="1.994419145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:05.974642484 +0000 UTC m=+484.350239232" watchObservedRunningTime="2026-11-03 20:05:05.976318242 +0000 UTC m=+484.351914992" Nov 03 20:05:05.999261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:05.996090 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-djrjk" podStartSLOduration=1.887112373 podStartE2EDuration="3.996060911s" totalImagesPullingTime="2.108948538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:05.99453158 +0000 UTC m=+484.370128327" watchObservedRunningTime="2026-11-03 20:05:05.996060911 +0000 UTC m=+484.371657659" Nov 03 20:05:06.041948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:06.041914 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-djrjk"] Nov 03 20:05:06.145732 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:05:06.145698 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:05:07.968784 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:07.968728 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-djrjk" podUID="c4bdbb41-d890-4509-b6c2-3e126f4f45a0" containerName="authorino" containerID="cri-o://4d3188a57ac587d8c4107bf3f5ef8d4ae722d014bf093a0bf09e98bf53f5da52" gracePeriod=30 Nov 03 20:05:08.208816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:08.208784 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-djrjk" Nov 03 20:05:08.346300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:08.346268 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4bdbb41-d890-4509-b6c2-3e126f4f45a0-kube-api-access-n4rfl\" (UniqueName: \"kubernetes.io/projected/c4bdbb41-d890-4509-b6c2-3e126f4f45a0-kube-api-access-n4rfl\") pod \"c4bdbb41-d890-4509-b6c2-3e126f4f45a0\" (UID: \"c4bdbb41-d890-4509-b6c2-3e126f4f45a0\") " Nov 03 20:05:08.348417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:08.348382 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bdbb41-d890-4509-b6c2-3e126f4f45a0-kube-api-access-n4rfl" pod "c4bdbb41-d890-4509-b6c2-3e126f4f45a0" (UID: "c4bdbb41-d890-4509-b6c2-3e126f4f45a0"). InnerVolumeSpecName "kube-api-access-n4rfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:08.447443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:08.447405 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4rfl\" (UniqueName: \"kubernetes.io/projected/c4bdbb41-d890-4509-b6c2-3e126f4f45a0-kube-api-access-n4rfl\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:08.974337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:08.974303 4457 generic.go:356] "Generic (PLEG): container finished" podID="c4bdbb41-d890-4509-b6c2-3e126f4f45a0" containerID="4d3188a57ac587d8c4107bf3f5ef8d4ae722d014bf093a0bf09e98bf53f5da52" exitCode=0 Nov 03 20:05:08.974810 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:08.974350 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-djrjk" Nov 03 20:05:08.974810 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:08.974368 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-djrjk" event={"ID":"c4bdbb41-d890-4509-b6c2-3e126f4f45a0","Type":"ContainerDied","Data":"4d3188a57ac587d8c4107bf3f5ef8d4ae722d014bf093a0bf09e98bf53f5da52"} Nov 03 20:05:08.974810 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:08.974426 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-djrjk" event={"ID":"c4bdbb41-d890-4509-b6c2-3e126f4f45a0","Type":"ContainerDied","Data":"7887d627d5f0e73fc433f9f0acd8d24e44f19175fa53b8962780d96242dc2f22"} Nov 03 20:05:08.974810 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:08.974443 4457 scope.go:122] "RemoveContainer" containerID="4d3188a57ac587d8c4107bf3f5ef8d4ae722d014bf093a0bf09e98bf53f5da52" Nov 03 20:05:08.984104 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:08.984078 4457 scope.go:122] "RemoveContainer" containerID="4d3188a57ac587d8c4107bf3f5ef8d4ae722d014bf093a0bf09e98bf53f5da52" Nov 03 20:05:08.984412 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:05:08.984374 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d3188a57ac587d8c4107bf3f5ef8d4ae722d014bf093a0bf09e98bf53f5da52\": container with ID starting with 4d3188a57ac587d8c4107bf3f5ef8d4ae722d014bf093a0bf09e98bf53f5da52 not found: ID does not exist" containerID="4d3188a57ac587d8c4107bf3f5ef8d4ae722d014bf093a0bf09e98bf53f5da52" Nov 03 20:05:08.984480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:08.984422 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d3188a57ac587d8c4107bf3f5ef8d4ae722d014bf093a0bf09e98bf53f5da52"} err="failed to get container status \"4d3188a57ac587d8c4107bf3f5ef8d4ae722d014bf093a0bf09e98bf53f5da52\": rpc error: code = NotFound desc = could not find container \"4d3188a57ac587d8c4107bf3f5ef8d4ae722d014bf093a0bf09e98bf53f5da52\": container with ID starting with 4d3188a57ac587d8c4107bf3f5ef8d4ae722d014bf093a0bf09e98bf53f5da52 not found: ID does not exist" Nov 03 20:05:08.997424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:08.997375 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-djrjk"] Nov 03 20:05:09.002286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:09.002260 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-djrjk"] Nov 03 20:05:09.754867 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:09.754836 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4bdbb41-d890-4509-b6c2-3e126f4f45a0" path="/var/lib/kubelet/pods/c4bdbb41-d890-4509-b6c2-3e126f4f45a0/volumes" Nov 03 20:05:12.995790 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:12.995756 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" event={"ID":"fa3f12f8-4195-46f0-8e87-f1da7e8400f7","Type":"ContainerStarted","Data":"d484ce1e850620b8769b3fc777704daa3828582be22c87c73d774e4369acf9e9"} Nov 03 20:05:12.996197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:12.995877 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" Nov 03 20:05:13.014845 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:13.014796 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" podStartSLOduration=2.1296105 podStartE2EDuration="12.014782034s" totalImagesPullingTime="9.885171534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:13.014780907 +0000 UTC m=+491.390377656" watchObservedRunningTime="2026-11-03 20:05:13.014782034 +0000 UTC m=+491.390378811" Nov 03 20:05:17.319420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:17.319327 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xg8rg"] Nov 03 20:05:17.319853 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:17.319520 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" podUID="fa3f12f8-4195-46f0-8e87-f1da7e8400f7" containerName="limitador" containerID="cri-o://d484ce1e850620b8769b3fc777704daa3828582be22c87c73d774e4369acf9e9" gracePeriod=30 Nov 03 20:05:17.321536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:17.321512 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" Nov 03 20:05:17.870190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:17.870165 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" Nov 03 20:05:18.020158 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.020066 4457 generic.go:356] "Generic (PLEG): container finished" podID="fa3f12f8-4195-46f0-8e87-f1da7e8400f7" containerID="d484ce1e850620b8769b3fc777704daa3828582be22c87c73d774e4369acf9e9" exitCode=0 Nov 03 20:05:18.020158 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.020116 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" Nov 03 20:05:18.020398 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.020119 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" event={"ID":"fa3f12f8-4195-46f0-8e87-f1da7e8400f7","Type":"ContainerDied","Data":"d484ce1e850620b8769b3fc777704daa3828582be22c87c73d774e4369acf9e9"} Nov 03 20:05:18.020398 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.020246 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-xg8rg" event={"ID":"fa3f12f8-4195-46f0-8e87-f1da7e8400f7","Type":"ContainerDied","Data":"5181ebbe33de23acc749bc86e0a13d489f2affdabf84d8f36f098628929cb0b5"} Nov 03 20:05:18.020398 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.020270 4457 scope.go:122] "RemoveContainer" containerID="d484ce1e850620b8769b3fc777704daa3828582be22c87c73d774e4369acf9e9" Nov 03 20:05:18.026236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.026216 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-config-file\" (UniqueName: \"kubernetes.io/configmap/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-config-file\") pod \"fa3f12f8-4195-46f0-8e87-f1da7e8400f7\" (UID: \"fa3f12f8-4195-46f0-8e87-f1da7e8400f7\") " Nov 03 20:05:18.026348 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.026295 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-kube-api-access-w2s9r\" (UniqueName: \"kubernetes.io/projected/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-kube-api-access-w2s9r\") pod \"fa3f12f8-4195-46f0-8e87-f1da7e8400f7\" (UID: \"fa3f12f8-4195-46f0-8e87-f1da7e8400f7\") " Nov 03 20:05:18.026594 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.026575 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-config-file" pod "fa3f12f8-4195-46f0-8e87-f1da7e8400f7" (UID: "fa3f12f8-4195-46f0-8e87-f1da7e8400f7"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:05:18.028215 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.028188 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-kube-api-access-w2s9r" pod "fa3f12f8-4195-46f0-8e87-f1da7e8400f7" (UID: "fa3f12f8-4195-46f0-8e87-f1da7e8400f7"). InnerVolumeSpecName "kube-api-access-w2s9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:18.029209 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.029182 4457 scope.go:122] "RemoveContainer" containerID="d484ce1e850620b8769b3fc777704daa3828582be22c87c73d774e4369acf9e9" Nov 03 20:05:18.029466 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:05:18.029442 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d484ce1e850620b8769b3fc777704daa3828582be22c87c73d774e4369acf9e9\": container with ID starting with d484ce1e850620b8769b3fc777704daa3828582be22c87c73d774e4369acf9e9 not found: ID does not exist" containerID="d484ce1e850620b8769b3fc777704daa3828582be22c87c73d774e4369acf9e9" Nov 03 20:05:18.029562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.029471 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d484ce1e850620b8769b3fc777704daa3828582be22c87c73d774e4369acf9e9"} err="failed to get container status \"d484ce1e850620b8769b3fc777704daa3828582be22c87c73d774e4369acf9e9\": rpc error: code = NotFound desc = could not find container \"d484ce1e850620b8769b3fc777704daa3828582be22c87c73d774e4369acf9e9\": container with ID starting with d484ce1e850620b8769b3fc777704daa3828582be22c87c73d774e4369acf9e9 not found: ID does not exist" Nov 03 20:05:18.127189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.127160 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2s9r\" (UniqueName: \"kubernetes.io/projected/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-kube-api-access-w2s9r\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:18.127189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.127186 4457 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fa3f12f8-4195-46f0-8e87-f1da7e8400f7-config-file\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:18.343255 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.343223 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xg8rg"] Nov 03 20:05:18.347607 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:18.347585 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-xg8rg"] Nov 03 20:05:19.754995 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:19.754962 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa3f12f8-4195-46f0-8e87-f1da7e8400f7" path="/var/lib/kubelet/pods/fa3f12f8-4195-46f0-8e87-f1da7e8400f7/volumes" Nov 03 20:05:24.075655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.075618 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z"] Nov 03 20:05:24.076038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.075961 4457 cpu_manager.go:414] "containerMap: removing container" podUID="fa3f12f8-4195-46f0-8e87-f1da7e8400f7" containerName="limitador" Nov 03 20:05:24.076038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.075972 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa3f12f8-4195-46f0-8e87-f1da7e8400f7" containerName="limitador" Nov 03 20:05:24.076038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.075988 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c4bdbb41-d890-4509-b6c2-3e126f4f45a0" containerName="authorino" Nov 03 20:05:24.076038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.075993 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4bdbb41-d890-4509-b6c2-3e126f4f45a0" containerName="authorino" Nov 03 20:05:24.076167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.076064 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa3f12f8-4195-46f0-8e87-f1da7e8400f7" containerName="limitador" Nov 03 20:05:24.076167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.076074 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4bdbb41-d890-4509-b6c2-3e126f4f45a0" containerName="authorino" Nov 03 20:05:24.080467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.080444 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z" Nov 03 20:05:24.085311 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.085290 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 20:05:24.086583 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.086542 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 20:05:24.086710 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.086583 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-rxp2s\"" Nov 03 20:05:24.088232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.088195 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z"] Nov 03 20:05:24.088502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.088480 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 20:05:24.180639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.180601 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/704584c9-f1c3-4968-bffe-92125ac2a96c-data\") pod \"postgres-7cdbf6dc44-g6q2z\" (UID: \"704584c9-f1c3-4968-bffe-92125ac2a96c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z" Nov 03 20:05:24.180824 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.180744 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnd87\" (UniqueName: \"kubernetes.io/projected/704584c9-f1c3-4968-bffe-92125ac2a96c-kube-api-access-gnd87\") pod \"postgres-7cdbf6dc44-g6q2z\" (UID: \"704584c9-f1c3-4968-bffe-92125ac2a96c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z" Nov 03 20:05:24.281627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.281594 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnd87\" (UniqueName: \"kubernetes.io/projected/704584c9-f1c3-4968-bffe-92125ac2a96c-kube-api-access-gnd87\") pod \"postgres-7cdbf6dc44-g6q2z\" (UID: \"704584c9-f1c3-4968-bffe-92125ac2a96c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z" Nov 03 20:05:24.281828 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.281659 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/704584c9-f1c3-4968-bffe-92125ac2a96c-data\") pod \"postgres-7cdbf6dc44-g6q2z\" (UID: \"704584c9-f1c3-4968-bffe-92125ac2a96c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z" Nov 03 20:05:24.282023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.282000 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/704584c9-f1c3-4968-bffe-92125ac2a96c-data\") pod \"postgres-7cdbf6dc44-g6q2z\" (UID: \"704584c9-f1c3-4968-bffe-92125ac2a96c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z" Nov 03 20:05:24.294214 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.294187 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnd87\" (UniqueName: \"kubernetes.io/projected/704584c9-f1c3-4968-bffe-92125ac2a96c-kube-api-access-gnd87\") pod \"postgres-7cdbf6dc44-g6q2z\" (UID: \"704584c9-f1c3-4968-bffe-92125ac2a96c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z" Nov 03 20:05:24.394470 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.394334 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z" Nov 03 20:05:24.666669 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:24.666641 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z"] Nov 03 20:05:24.669373 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:05:24.669346 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704584c9_f1c3_4968_bffe_92125ac2a96c.slice/crio-9cd59b26ccfe21c1e99f3a093f9b2aff49bfd7f04240b18c933c3cd54165b5f3 WatchSource:0}: Error finding container 9cd59b26ccfe21c1e99f3a093f9b2aff49bfd7f04240b18c933c3cd54165b5f3: Status 404 returned error can't find the container with id 9cd59b26ccfe21c1e99f3a093f9b2aff49bfd7f04240b18c933c3cd54165b5f3 Nov 03 20:05:25.060281 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:25.060242 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z" event={"ID":"704584c9-f1c3-4968-bffe-92125ac2a96c","Type":"ContainerStarted","Data":"9cd59b26ccfe21c1e99f3a093f9b2aff49bfd7f04240b18c933c3cd54165b5f3"} Nov 03 20:05:31.094756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:31.094720 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z" event={"ID":"704584c9-f1c3-4968-bffe-92125ac2a96c","Type":"ContainerStarted","Data":"8466f02238558a4d50f43ec3ed9f2e2d034282a43fbd8301c2bb20f45aa255c5"} Nov 03 20:05:31.095196 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:31.094854 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z" Nov 03 20:05:31.114601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:31.114547 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z" podStartSLOduration=1.573195202 podStartE2EDuration="7.114534284s" totalImagesPullingTime="5.541339082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:31.11312484 +0000 UTC m=+509.488721589" watchObservedRunningTime="2026-11-03 20:05:31.114534284 +0000 UTC m=+509.490131057" Nov 03 20:05:36.143292 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:05:36.143257 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:05:37.128884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:37.128859 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-g6q2z" Nov 03 20:05:44.155934 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.155899 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-8m427"] Nov 03 20:05:44.175180 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.175147 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-8m427"] Nov 03 20:05:44.175359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.175276 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:44.182714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.182688 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 20:05:44.188512 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.187573 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 20:05:44.188512 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.188226 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-64wg7\"" Nov 03 20:05:44.188512 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.188431 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 20:05:44.188512 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.188460 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 20:05:44.188512 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.188510 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 20:05:44.240513 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.240480 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-metrics-certs\") pod \"maas-controller-579cf7c6d-8m427\" (UID: \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\") " pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:44.240666 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.240593 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-webhook-cert\") pod \"maas-controller-579cf7c6d-8m427\" (UID: \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\") " pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:44.240666 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.240622 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb9p5\" (UniqueName: \"kubernetes.io/projected/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-kube-api-access-cb9p5\") pod \"maas-controller-579cf7c6d-8m427\" (UID: \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\") " pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:44.341661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.341627 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-webhook-cert\") pod \"maas-controller-579cf7c6d-8m427\" (UID: \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\") " pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:44.341661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.341663 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb9p5\" (UniqueName: \"kubernetes.io/projected/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-kube-api-access-cb9p5\") pod \"maas-controller-579cf7c6d-8m427\" (UID: \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\") " pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:44.341890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.341692 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-metrics-certs\") pod \"maas-controller-579cf7c6d-8m427\" (UID: \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\") " pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:44.344172 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.344144 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-webhook-cert\") pod \"maas-controller-579cf7c6d-8m427\" (UID: \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\") " pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:44.344285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.344196 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-metrics-certs\") pod \"maas-controller-579cf7c6d-8m427\" (UID: \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\") " pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:44.356203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.356180 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb9p5\" (UniqueName: \"kubernetes.io/projected/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-kube-api-access-cb9p5\") pod \"maas-controller-579cf7c6d-8m427\" (UID: \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\") " pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:44.493558 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.493482 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:44.494373 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.494352 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-69l6m"] Nov 03 20:05:44.496583 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.496564 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:05:44.509608 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.509573 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-69l6m"] Nov 03 20:05:44.544654 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.544020 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-webhook-cert\") pod \"maas-controller-6dd9f77dd9-69l6m\" (UID: \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\") " pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:05:44.544654 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.544076 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf7rc\" (UniqueName: \"kubernetes.io/projected/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-kube-api-access-qf7rc\") pod \"maas-controller-6dd9f77dd9-69l6m\" (UID: \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\") " pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:05:44.544654 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.544120 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-metrics-certs\") pod \"maas-controller-6dd9f77dd9-69l6m\" (UID: \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\") " pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:05:44.644507 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.644473 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-metrics-certs\") pod \"maas-controller-6dd9f77dd9-69l6m\" (UID: \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\") " pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:05:44.644507 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.644509 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-webhook-cert\") pod \"maas-controller-6dd9f77dd9-69l6m\" (UID: \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\") " pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:05:44.644721 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.644550 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf7rc\" (UniqueName: \"kubernetes.io/projected/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-kube-api-access-qf7rc\") pod \"maas-controller-6dd9f77dd9-69l6m\" (UID: \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\") " pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:05:44.646914 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.646892 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-metrics-certs\") pod \"maas-controller-6dd9f77dd9-69l6m\" (UID: \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\") " pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:05:44.646914 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.646910 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-webhook-cert\") pod \"maas-controller-6dd9f77dd9-69l6m\" (UID: \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\") " pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:05:44.656826 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.656802 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf7rc\" (UniqueName: \"kubernetes.io/projected/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-kube-api-access-qf7rc\") pod \"maas-controller-6dd9f77dd9-69l6m\" (UID: \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\") " pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:05:44.773198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.773175 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-8m427"] Nov 03 20:05:44.774666 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:05:44.774637 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7e0a63_a7f3_4d64_8d85_7caf7a680c89.slice/crio-0abf0c39cadd761ff4a1e0b8af4f70061a4839b6f2f0247c82823a897a87f73a WatchSource:0}: Error finding container 0abf0c39cadd761ff4a1e0b8af4f70061a4839b6f2f0247c82823a897a87f73a: Status 404 returned error can't find the container with id 0abf0c39cadd761ff4a1e0b8af4f70061a4839b6f2f0247c82823a897a87f73a Nov 03 20:05:44.826090 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:44.826057 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:05:45.103023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:45.103001 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-69l6m"] Nov 03 20:05:45.104882 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:05:45.104856 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb0a4bf_7a65_40ea_9a42_6d1d89c76645.slice/crio-f31fd8fbd1ceeecb37218ec94d1c4007b90c07fc020c6b38f3d22d72541644d6 WatchSource:0}: Error finding container f31fd8fbd1ceeecb37218ec94d1c4007b90c07fc020c6b38f3d22d72541644d6: Status 404 returned error can't find the container with id f31fd8fbd1ceeecb37218ec94d1c4007b90c07fc020c6b38f3d22d72541644d6 Nov 03 20:05:45.161266 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:45.161228 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" event={"ID":"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645","Type":"ContainerStarted","Data":"f31fd8fbd1ceeecb37218ec94d1c4007b90c07fc020c6b38f3d22d72541644d6"} Nov 03 20:05:45.162525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:45.162501 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-8m427" event={"ID":"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89","Type":"ContainerStarted","Data":"0abf0c39cadd761ff4a1e0b8af4f70061a4839b6f2f0247c82823a897a87f73a"} Nov 03 20:05:48.185749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.185700 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" event={"ID":"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645","Type":"ContainerStarted","Data":"4cacf177178e5818c08b46ac32f56a2bf1c417ae930a3fa3388b54abd1e4dc95"} Nov 03 20:05:48.186258 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.185792 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:05:48.187672 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.187640 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-8m427" event={"ID":"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89","Type":"ContainerStarted","Data":"d3abce403e920e0cb94b9c24a15a66350a282702c1570d49153ed9b90805b3cb"} Nov 03 20:05:48.187965 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.187947 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:48.207215 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.207165 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" podStartSLOduration=1.435371073 podStartE2EDuration="4.20715438s" totalImagesPullingTime="2.771783307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:48.20596548 +0000 UTC m=+526.581562227" watchObservedRunningTime="2026-11-03 20:05:48.20715438 +0000 UTC m=+526.582751126" Nov 03 20:05:48.227893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.227851 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-8m427" podStartSLOduration=1.234106935 podStartE2EDuration="4.227839542s" totalImagesPullingTime="2.993732607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:05:48.227155975 +0000 UTC m=+526.602752723" watchObservedRunningTime="2026-11-03 20:05:48.227839542 +0000 UTC m=+526.603436289" Nov 03 20:05:48.912723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.912693 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv"] Nov 03 20:05:48.916236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.916219 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:48.923698 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.923626 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 20:05:48.925312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.925156 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 20:05:48.925312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.925231 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 20:05:48.925312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.925267 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-mlplc\"" Nov 03 20:05:48.935051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.935030 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv"] Nov 03 20:05:48.985876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.985839 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cbnpv\" (UID: \"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:48.986033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.985892 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cbnpv\" (UID: \"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:48.986033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.985919 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cbnpv\" (UID: \"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:48.986033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:48.985983 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpz95\" (UniqueName: \"kubernetes.io/projected/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-kube-api-access-bpz95\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cbnpv\" (UID: \"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:49.087293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:49.087255 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cbnpv\" (UID: \"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:49.087499 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:49.087316 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cbnpv\" (UID: \"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:49.087499 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:05:49.087425 4457 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 03 20:05:49.087499 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:05:49.087496 4457 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-tls-cert podName:4c6a37ee-a6e8-4345-990b-82ef4d3d3a77 nodeName:}" failed. No retries permitted until 2026-11-03 20:05:49.587477556 +0000 UTC m=+527.963074295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" (UID: "4c6a37ee-a6e8-4345-990b-82ef4d3d3a77") : secret "usage-logs-tenancy-proxy-tls" not found Nov 03 20:05:49.087679 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:49.087517 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cbnpv\" (UID: \"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:49.087679 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:49.087561 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpz95\" (UniqueName: \"kubernetes.io/projected/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-kube-api-access-bpz95\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cbnpv\" (UID: \"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:49.088046 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:49.088023 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cbnpv\" (UID: \"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:49.088585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:49.088565 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cbnpv\" (UID: \"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:49.100229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:49.100203 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpz95\" (UniqueName: \"kubernetes.io/projected/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-kube-api-access-bpz95\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cbnpv\" (UID: \"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:49.592355 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:49.592318 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cbnpv\" (UID: \"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:49.594760 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:49.594741 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4c6a37ee-a6e8-4345-990b-82ef4d3d3a77-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cbnpv\" (UID: \"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:49.831963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:49.831928 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:05:50.117713 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:50.117683 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv"] Nov 03 20:05:50.119594 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:05:50.119567 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6a37ee_a6e8_4345_990b_82ef4d3d3a77.slice/crio-34ab07b1632ced42769f5187f5e9d516c6e4a1653f213b1e65c5726d03e05c85 WatchSource:0}: Error finding container 34ab07b1632ced42769f5187f5e9d516c6e4a1653f213b1e65c5726d03e05c85: Status 404 returned error can't find the container with id 34ab07b1632ced42769f5187f5e9d516c6e4a1653f213b1e65c5726d03e05c85 Nov 03 20:05:50.199298 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:50.199264 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" event={"ID":"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77","Type":"ContainerStarted","Data":"34ab07b1632ced42769f5187f5e9d516c6e4a1653f213b1e65c5726d03e05c85"} Nov 03 20:05:57.523412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.522737 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c"] Nov 03 20:05:57.530418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.527991 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" Nov 03 20:05:57.544074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.544048 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c"] Nov 03 20:05:57.549356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.549332 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mq8tg"] Nov 03 20:05:57.553629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.553610 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" Nov 03 20:05:57.603678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.599875 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vbqzr\"" Nov 03 20:05:57.603678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.600028 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-4s45t\"" Nov 03 20:05:57.603678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.600089 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 20:05:57.603678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.600209 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 03 20:05:57.607134 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.607109 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mq8tg"] Nov 03 20:05:57.607840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.607803 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5bk7c\" (UID: \"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" Nov 03 20:05:57.609306 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.608017 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52zxh\" (UniqueName: \"kubernetes.io/projected/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-kube-api-access-52zxh\") pod \"llmisvc-controller-manager-869d5dcc5d-5bk7c\" (UID: \"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" Nov 03 20:05:57.710146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.710109 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e100b28-7be2-4dcf-abbc-3532f79ff178-tls-certs\") pod \"model-serving-api-75bd89fdfc-mq8tg\" (UID: \"2e100b28-7be2-4dcf-abbc-3532f79ff178\") " pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" Nov 03 20:05:57.710294 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.710171 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5bk7c\" (UID: \"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" Nov 03 20:05:57.710294 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.710256 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52zxh\" (UniqueName: \"kubernetes.io/projected/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-kube-api-access-52zxh\") pod \"llmisvc-controller-manager-869d5dcc5d-5bk7c\" (UID: \"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" Nov 03 20:05:57.710374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.710297 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gtxb\" (UniqueName: \"kubernetes.io/projected/2e100b28-7be2-4dcf-abbc-3532f79ff178-kube-api-access-4gtxb\") pod \"model-serving-api-75bd89fdfc-mq8tg\" (UID: \"2e100b28-7be2-4dcf-abbc-3532f79ff178\") " pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" Nov 03 20:05:57.712516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.712500 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-5bk7c\" (UID: \"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" Nov 03 20:05:57.724359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.724329 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52zxh\" (UniqueName: \"kubernetes.io/projected/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-kube-api-access-52zxh\") pod \"llmisvc-controller-manager-869d5dcc5d-5bk7c\" (UID: \"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" Nov 03 20:05:57.811017 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.810987 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e100b28-7be2-4dcf-abbc-3532f79ff178-tls-certs\") pod \"model-serving-api-75bd89fdfc-mq8tg\" (UID: \"2e100b28-7be2-4dcf-abbc-3532f79ff178\") " pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" Nov 03 20:05:57.811165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.811073 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gtxb\" (UniqueName: \"kubernetes.io/projected/2e100b28-7be2-4dcf-abbc-3532f79ff178-kube-api-access-4gtxb\") pod \"model-serving-api-75bd89fdfc-mq8tg\" (UID: \"2e100b28-7be2-4dcf-abbc-3532f79ff178\") " pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" Nov 03 20:05:57.813451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.813423 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e100b28-7be2-4dcf-abbc-3532f79ff178-tls-certs\") pod \"model-serving-api-75bd89fdfc-mq8tg\" (UID: \"2e100b28-7be2-4dcf-abbc-3532f79ff178\") " pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" Nov 03 20:05:57.825538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.825512 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gtxb\" (UniqueName: \"kubernetes.io/projected/2e100b28-7be2-4dcf-abbc-3532f79ff178-kube-api-access-4gtxb\") pod \"model-serving-api-75bd89fdfc-mq8tg\" (UID: \"2e100b28-7be2-4dcf-abbc-3532f79ff178\") " pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" Nov 03 20:05:57.848492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.848468 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" Nov 03 20:05:57.925350 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:57.925318 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" Nov 03 20:05:58.122794 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:58.122767 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c"] Nov 03 20:05:58.124736 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:05:58.124711 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poded8f5019_f8d7_40ab_b31a_cd36b9f9b67f.slice/crio-12644fed7e587c81cd52a33986a6ddf7b1e5e6dcd21a38bce82e6e6400e40032 WatchSource:0}: Error finding container 12644fed7e587c81cd52a33986a6ddf7b1e5e6dcd21a38bce82e6e6400e40032: Status 404 returned error can't find the container with id 12644fed7e587c81cd52a33986a6ddf7b1e5e6dcd21a38bce82e6e6400e40032 Nov 03 20:05:58.200077 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:58.200048 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-mq8tg"] Nov 03 20:05:58.202552 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:05:58.202523 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e100b28_7be2_4dcf_abbc_3532f79ff178.slice/crio-6adb32821ebe3d19bad57241899e6543da2bb84091a21c7d3d2ff5d60988adc3 WatchSource:0}: Error finding container 6adb32821ebe3d19bad57241899e6543da2bb84091a21c7d3d2ff5d60988adc3: Status 404 returned error can't find the container with id 6adb32821ebe3d19bad57241899e6543da2bb84091a21c7d3d2ff5d60988adc3 Nov 03 20:05:58.240028 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:58.239992 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" event={"ID":"2e100b28-7be2-4dcf-abbc-3532f79ff178","Type":"ContainerStarted","Data":"6adb32821ebe3d19bad57241899e6543da2bb84091a21c7d3d2ff5d60988adc3"} Nov 03 20:05:58.241220 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:58.241198 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" event={"ID":"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f","Type":"ContainerStarted","Data":"12644fed7e587c81cd52a33986a6ddf7b1e5e6dcd21a38bce82e6e6400e40032"} Nov 03 20:05:59.197228 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.195699 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:59.197228 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.196708 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:05:59.277589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.277554 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-8m427"] Nov 03 20:05:59.278082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.278026 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-8m427" podUID="2b7e0a63-a7f3-4d64-8d85-7caf7a680c89" containerName="manager" containerID="cri-o://d3abce403e920e0cb94b9c24a15a66350a282702c1570d49153ed9b90805b3cb" gracePeriod=30 Nov 03 20:05:59.586994 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.586954 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:05:59.736000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.735963 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-webhook-cert\") pod \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\" (UID: \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\") " Nov 03 20:05:59.736165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.736033 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-metrics-certs\") pod \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\" (UID: \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\") " Nov 03 20:05:59.736165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.736082 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-kube-api-access-cb9p5\" (UniqueName: \"kubernetes.io/projected/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-kube-api-access-cb9p5\") pod \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\" (UID: \"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89\") " Nov 03 20:05:59.739213 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.739142 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-kube-api-access-cb9p5" pod "2b7e0a63-a7f3-4d64-8d85-7caf7a680c89" (UID: "2b7e0a63-a7f3-4d64-8d85-7caf7a680c89"). InnerVolumeSpecName "kube-api-access-cb9p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:05:59.740917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.740891 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-metrics-certs" pod "2b7e0a63-a7f3-4d64-8d85-7caf7a680c89" (UID: "2b7e0a63-a7f3-4d64-8d85-7caf7a680c89"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:59.741230 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.741102 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-webhook-cert" pod "2b7e0a63-a7f3-4d64-8d85-7caf7a680c89" (UID: "2b7e0a63-a7f3-4d64-8d85-7caf7a680c89"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:05:59.837655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.837615 4457 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-webhook-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:59.837655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.837653 4457 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-metrics-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:05:59.837914 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:05:59.837670 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb9p5\" (UniqueName: \"kubernetes.io/projected/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89-kube-api-access-cb9p5\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:00.265555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:00.265512 4457 generic.go:356] "Generic (PLEG): container finished" podID="2b7e0a63-a7f3-4d64-8d85-7caf7a680c89" containerID="d3abce403e920e0cb94b9c24a15a66350a282702c1570d49153ed9b90805b3cb" exitCode=0 Nov 03 20:06:00.266023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:00.265596 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-8m427" Nov 03 20:06:00.266023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:00.265609 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-8m427" event={"ID":"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89","Type":"ContainerDied","Data":"d3abce403e920e0cb94b9c24a15a66350a282702c1570d49153ed9b90805b3cb"} Nov 03 20:06:00.266023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:00.265660 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-8m427" event={"ID":"2b7e0a63-a7f3-4d64-8d85-7caf7a680c89","Type":"ContainerDied","Data":"0abf0c39cadd761ff4a1e0b8af4f70061a4839b6f2f0247c82823a897a87f73a"} Nov 03 20:06:00.266023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:00.265689 4457 scope.go:122] "RemoveContainer" containerID="d3abce403e920e0cb94b9c24a15a66350a282702c1570d49153ed9b90805b3cb" Nov 03 20:06:00.277136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:00.277107 4457 scope.go:122] "RemoveContainer" containerID="d3abce403e920e0cb94b9c24a15a66350a282702c1570d49153ed9b90805b3cb" Nov 03 20:06:00.277584 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:06:00.277555 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3abce403e920e0cb94b9c24a15a66350a282702c1570d49153ed9b90805b3cb\": container with ID starting with d3abce403e920e0cb94b9c24a15a66350a282702c1570d49153ed9b90805b3cb not found: ID does not exist" containerID="d3abce403e920e0cb94b9c24a15a66350a282702c1570d49153ed9b90805b3cb" Nov 03 20:06:00.277721 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:00.277589 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3abce403e920e0cb94b9c24a15a66350a282702c1570d49153ed9b90805b3cb"} err="failed to get container status \"d3abce403e920e0cb94b9c24a15a66350a282702c1570d49153ed9b90805b3cb\": rpc error: code = NotFound desc = could not find container \"d3abce403e920e0cb94b9c24a15a66350a282702c1570d49153ed9b90805b3cb\": container with ID starting with d3abce403e920e0cb94b9c24a15a66350a282702c1570d49153ed9b90805b3cb not found: ID does not exist" Nov 03 20:06:00.283897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:00.283874 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-8m427"] Nov 03 20:06:00.290069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:00.290043 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-8m427"] Nov 03 20:06:01.273434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:01.273380 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" event={"ID":"2e100b28-7be2-4dcf-abbc-3532f79ff178","Type":"ContainerStarted","Data":"eda8461a125a8b71df045fdfc62ab93f2818005317a92693e63926dde91f64cf"} Nov 03 20:06:01.273747 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:01.273509 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" Nov 03 20:06:01.293647 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:01.293604 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" podStartSLOduration=2.052659412 podStartE2EDuration="4.29359082s" totalImagesPullingTime="2.240931408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:01.292059315 +0000 UTC m=+539.667656062" watchObservedRunningTime="2026-11-03 20:06:01.29359082 +0000 UTC m=+539.669187623" Nov 03 20:06:01.756272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:01.756232 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b7e0a63-a7f3-4d64-8d85-7caf7a680c89" path="/var/lib/kubelet/pods/2b7e0a63-a7f3-4d64-8d85-7caf7a680c89/volumes" Nov 03 20:06:02.280319 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:02.280284 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" event={"ID":"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f","Type":"ContainerStarted","Data":"06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4"} Nov 03 20:06:02.280746 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:02.280373 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" Nov 03 20:06:02.301373 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:02.301323 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" podStartSLOduration=2.205874588 podStartE2EDuration="5.301310375s" totalImagesPullingTime="3.095435787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:02.298779525 +0000 UTC m=+540.674376298" watchObservedRunningTime="2026-11-03 20:06:02.301310375 +0000 UTC m=+540.676907122" Nov 03 20:06:06.149622 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:06:06.149581 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:06:12.284967 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:12.284934 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" Nov 03 20:06:12.362646 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:12.362610 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" event={"ID":"4c6a37ee-a6e8-4345-990b-82ef4d3d3a77","Type":"ContainerStarted","Data":"c3262943382b146732728956e05c08a78ba3efe2105946f9aa07213093eacd36"} Nov 03 20:06:12.362855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:12.362818 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:06:12.383209 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:12.383157 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" podStartSLOduration=2.8674073890000003 podStartE2EDuration="24.383142223s" totalImagesPullingTime="21.515734834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:05:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:12.382029807 +0000 UTC m=+550.757626552" watchObservedRunningTime="2026-11-03 20:06:12.383142223 +0000 UTC m=+550.758738970" Nov 03 20:06:23.384124 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:23.384086 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cbnpv" Nov 03 20:06:32.273312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.273221 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bbgr2"] Nov 03 20:06:32.273747 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.273621 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2b7e0a63-a7f3-4d64-8d85-7caf7a680c89" containerName="manager" Nov 03 20:06:32.273747 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.273633 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b7e0a63-a7f3-4d64-8d85-7caf7a680c89" containerName="manager" Nov 03 20:06:32.273747 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.273685 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b7e0a63-a7f3-4d64-8d85-7caf7a680c89" containerName="manager" Nov 03 20:06:32.275608 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.275587 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bbgr2" Nov 03 20:06:32.283448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.283007 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bbgr2"] Nov 03 20:06:32.361831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.361791 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bbgr2"] Nov 03 20:06:32.362042 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:06:32.362024 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jph97], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-bbgr2" podUID="5264de60-fa40-4e5b-8e48-afe9fd6fa9fb" Nov 03 20:06:32.384862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.384816 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jph97\" (UniqueName: \"kubernetes.io/projected/5264de60-fa40-4e5b-8e48-afe9fd6fa9fb-kube-api-access-jph97\") pod \"authorino-6f6944bcdd-bbgr2\" (UID: \"5264de60-fa40-4e5b-8e48-afe9fd6fa9fb\") " pod="kuadrant-system/authorino-6f6944bcdd-bbgr2" Nov 03 20:06:32.464706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.464675 4457 generic.go:356] "Generic (PLEG): container finished" podID="93f7f2c5-96a8-4ccc-ad9b-3cc648f85802" containerID="1f41ec797c3adc0673604f6e6ebb6628f7b4cdb9537252d2a3d3baf32fe76223" exitCode=0 Nov 03 20:06:32.464893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.464758 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bbgr2" Nov 03 20:06:32.464893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.464752 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" event={"ID":"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802","Type":"ContainerDied","Data":"1f41ec797c3adc0673604f6e6ebb6628f7b4cdb9537252d2a3d3baf32fe76223"} Nov 03 20:06:32.465167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.465151 4457 scope.go:122] "RemoveContainer" containerID="1f41ec797c3adc0673604f6e6ebb6628f7b4cdb9537252d2a3d3baf32fe76223" Nov 03 20:06:32.466359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.466331 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:06:32.466471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.466423 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:06:32.466471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.466456 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:06:32.470622 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.470604 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bbgr2" Nov 03 20:06:32.486360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.486336 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jph97\" (UniqueName: \"kubernetes.io/projected/5264de60-fa40-4e5b-8e48-afe9fd6fa9fb-kube-api-access-jph97\") pod \"authorino-6f6944bcdd-bbgr2\" (UID: \"5264de60-fa40-4e5b-8e48-afe9fd6fa9fb\") " pod="kuadrant-system/authorino-6f6944bcdd-bbgr2" Nov 03 20:06:32.501018 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.500990 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jph97\" (UniqueName: \"kubernetes.io/projected/5264de60-fa40-4e5b-8e48-afe9fd6fa9fb-kube-api-access-jph97\") pod \"authorino-6f6944bcdd-bbgr2\" (UID: \"5264de60-fa40-4e5b-8e48-afe9fd6fa9fb\") " pod="kuadrant-system/authorino-6f6944bcdd-bbgr2" Nov 03 20:06:32.688035 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.688002 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5264de60-fa40-4e5b-8e48-afe9fd6fa9fb-kube-api-access-jph97\" (UniqueName: \"kubernetes.io/projected/5264de60-fa40-4e5b-8e48-afe9fd6fa9fb-kube-api-access-jph97\") pod \"5264de60-fa40-4e5b-8e48-afe9fd6fa9fb\" (UID: \"5264de60-fa40-4e5b-8e48-afe9fd6fa9fb\") " Nov 03 20:06:32.690268 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.690225 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5264de60-fa40-4e5b-8e48-afe9fd6fa9fb-kube-api-access-jph97" pod "5264de60-fa40-4e5b-8e48-afe9fd6fa9fb" (UID: "5264de60-fa40-4e5b-8e48-afe9fd6fa9fb"). InnerVolumeSpecName "kube-api-access-jph97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:32.697193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.697169 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-79b6656fd9-m22gh"] Nov 03 20:06:32.699596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.699579 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b6656fd9-m22gh" Nov 03 20:06:32.703636 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.703607 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 20:06:32.709085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.709057 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79b6656fd9-m22gh"] Nov 03 20:06:32.788823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.788736 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpf7t\" (UniqueName: \"kubernetes.io/projected/7e4e5eed-61cb-4444-b0f9-a398d67ab773-kube-api-access-bpf7t\") pod \"authorino-79b6656fd9-m22gh\" (UID: \"7e4e5eed-61cb-4444-b0f9-a398d67ab773\") " pod="kuadrant-system/authorino-79b6656fd9-m22gh" Nov 03 20:06:32.788823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.788782 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e4e5eed-61cb-4444-b0f9-a398d67ab773-tls-cert\") pod \"authorino-79b6656fd9-m22gh\" (UID: \"7e4e5eed-61cb-4444-b0f9-a398d67ab773\") " pod="kuadrant-system/authorino-79b6656fd9-m22gh" Nov 03 20:06:32.789037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.788938 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jph97\" (UniqueName: \"kubernetes.io/projected/5264de60-fa40-4e5b-8e48-afe9fd6fa9fb-kube-api-access-jph97\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:32.889361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.889314 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpf7t\" (UniqueName: \"kubernetes.io/projected/7e4e5eed-61cb-4444-b0f9-a398d67ab773-kube-api-access-bpf7t\") pod \"authorino-79b6656fd9-m22gh\" (UID: \"7e4e5eed-61cb-4444-b0f9-a398d67ab773\") " pod="kuadrant-system/authorino-79b6656fd9-m22gh" Nov 03 20:06:32.889361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.889362 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e4e5eed-61cb-4444-b0f9-a398d67ab773-tls-cert\") pod \"authorino-79b6656fd9-m22gh\" (UID: \"7e4e5eed-61cb-4444-b0f9-a398d67ab773\") " pod="kuadrant-system/authorino-79b6656fd9-m22gh" Nov 03 20:06:32.891920 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.891887 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e4e5eed-61cb-4444-b0f9-a398d67ab773-tls-cert\") pod \"authorino-79b6656fd9-m22gh\" (UID: \"7e4e5eed-61cb-4444-b0f9-a398d67ab773\") " pod="kuadrant-system/authorino-79b6656fd9-m22gh" Nov 03 20:06:32.901740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:32.901716 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpf7t\" (UniqueName: \"kubernetes.io/projected/7e4e5eed-61cb-4444-b0f9-a398d67ab773-kube-api-access-bpf7t\") pod \"authorino-79b6656fd9-m22gh\" (UID: \"7e4e5eed-61cb-4444-b0f9-a398d67ab773\") " pod="kuadrant-system/authorino-79b6656fd9-m22gh" Nov 03 20:06:33.011886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:33.011855 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b6656fd9-m22gh" Nov 03 20:06:33.308918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:33.308889 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-79b6656fd9-m22gh"] Nov 03 20:06:33.311070 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:06:33.311035 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e4e5eed_61cb_4444_b0f9_a398d67ab773.slice/crio-ead42e8404018ebfb5a36521b2b04dd8cc04bb749bfc70d5eb33380c21f8c661 WatchSource:0}: Error finding container ead42e8404018ebfb5a36521b2b04dd8cc04bb749bfc70d5eb33380c21f8c661: Status 404 returned error can't find the container with id ead42e8404018ebfb5a36521b2b04dd8cc04bb749bfc70d5eb33380c21f8c661 Nov 03 20:06:33.311199 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:33.311106 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" Nov 03 20:06:33.486253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:33.486160 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" event={"ID":"93f7f2c5-96a8-4ccc-ad9b-3cc648f85802","Type":"ContainerStarted","Data":"a940c5822787d319ed9b2859ff9c8171e1779b7142a9e6b41c6a3813f655dd06"} Nov 03 20:06:33.487436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:33.487405 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79b6656fd9-m22gh" event={"ID":"7e4e5eed-61cb-4444-b0f9-a398d67ab773","Type":"ContainerStarted","Data":"ead42e8404018ebfb5a36521b2b04dd8cc04bb749bfc70d5eb33380c21f8c661"} Nov 03 20:06:33.487563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:33.487438 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-bbgr2" Nov 03 20:06:33.542032 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:33.541998 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bbgr2"] Nov 03 20:06:33.546461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:33.546433 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-bbgr2"] Nov 03 20:06:33.771471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:33.755576 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5264de60-fa40-4e5b-8e48-afe9fd6fa9fb" path="/var/lib/kubelet/pods/5264de60-fa40-4e5b-8e48-afe9fd6fa9fb/volumes" Nov 03 20:06:33.811541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:33.811506 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:06:33.812605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:33.812579 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p7lrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Nov 03 20:06:33.812709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:33.812636 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" podUID="93f7f2c5-96a8-4ccc-ad9b-3cc648f85802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Nov 03 20:06:34.501473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:34.496940 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79b6656fd9-m22gh" event={"ID":"7e4e5eed-61cb-4444-b0f9-a398d67ab773","Type":"ContainerStarted","Data":"95dd97a6669086a8da2af5279fcb6f047d3cc297d9d4e7b3415372db608b008b"} Nov 03 20:06:34.518536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:34.518480 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-79b6656fd9-m22gh" podStartSLOduration=1.682976778 podStartE2EDuration="2.518463195s" totalImagesPullingTime="835.486417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:34.516958121 +0000 UTC m=+572.892554868" watchObservedRunningTime="2026-11-03 20:06:34.518463195 +0000 UTC m=+572.894059944" Nov 03 20:06:34.550597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:34.550553 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5jhck"] Nov 03 20:06:34.550897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:34.550867 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-5jhck" podUID="26273dd2-22e7-419d-bbc0-cf1f831ef5f8" containerName="authorino" containerID="cri-o://6a84c99cd862fccb71661bacec9899ecaeab348650d3dc606f7ed4c35866176c" gracePeriod=30 Nov 03 20:06:34.812413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:34.812037 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p7lrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Nov 03 20:06:34.812413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:34.812088 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" podUID="93f7f2c5-96a8-4ccc-ad9b-3cc648f85802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Nov 03 20:06:34.881427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:34.881259 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5jhck" Nov 03 20:06:34.907457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:34.906455 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26273dd2-22e7-419d-bbc0-cf1f831ef5f8-kube-api-access-chdfh\" (UniqueName: \"kubernetes.io/projected/26273dd2-22e7-419d-bbc0-cf1f831ef5f8-kube-api-access-chdfh\") pod \"26273dd2-22e7-419d-bbc0-cf1f831ef5f8\" (UID: \"26273dd2-22e7-419d-bbc0-cf1f831ef5f8\") " Nov 03 20:06:34.909629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:34.909587 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26273dd2-22e7-419d-bbc0-cf1f831ef5f8-kube-api-access-chdfh" pod "26273dd2-22e7-419d-bbc0-cf1f831ef5f8" (UID: "26273dd2-22e7-419d-bbc0-cf1f831ef5f8"). InnerVolumeSpecName "kube-api-access-chdfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:06:35.007147 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:35.007104 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chdfh\" (UniqueName: \"kubernetes.io/projected/26273dd2-22e7-419d-bbc0-cf1f831ef5f8-kube-api-access-chdfh\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:06:35.503737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:35.503641 4457 generic.go:356] "Generic (PLEG): container finished" podID="26273dd2-22e7-419d-bbc0-cf1f831ef5f8" containerID="6a84c99cd862fccb71661bacec9899ecaeab348650d3dc606f7ed4c35866176c" exitCode=0 Nov 03 20:06:35.503737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:35.503692 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5jhck" Nov 03 20:06:35.503737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:35.503727 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5jhck" event={"ID":"26273dd2-22e7-419d-bbc0-cf1f831ef5f8","Type":"ContainerDied","Data":"6a84c99cd862fccb71661bacec9899ecaeab348650d3dc606f7ed4c35866176c"} Nov 03 20:06:35.504223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:35.503772 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5jhck" event={"ID":"26273dd2-22e7-419d-bbc0-cf1f831ef5f8","Type":"ContainerDied","Data":"06c66bb9e3bf55588c6a77233d26b51b284e596aec74d3b0e8d3a9592a7c35b5"} Nov 03 20:06:35.504223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:35.503789 4457 scope.go:122] "RemoveContainer" containerID="6a84c99cd862fccb71661bacec9899ecaeab348650d3dc606f7ed4c35866176c" Nov 03 20:06:35.518540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:35.518512 4457 scope.go:122] "RemoveContainer" containerID="6a84c99cd862fccb71661bacec9899ecaeab348650d3dc606f7ed4c35866176c" Nov 03 20:06:35.519038 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:06:35.519017 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a84c99cd862fccb71661bacec9899ecaeab348650d3dc606f7ed4c35866176c\": container with ID starting with 6a84c99cd862fccb71661bacec9899ecaeab348650d3dc606f7ed4c35866176c not found: ID does not exist" containerID="6a84c99cd862fccb71661bacec9899ecaeab348650d3dc606f7ed4c35866176c" Nov 03 20:06:35.519119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:35.519045 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a84c99cd862fccb71661bacec9899ecaeab348650d3dc606f7ed4c35866176c"} err="failed to get container status \"6a84c99cd862fccb71661bacec9899ecaeab348650d3dc606f7ed4c35866176c\": rpc error: code = NotFound desc = could not find container \"6a84c99cd862fccb71661bacec9899ecaeab348650d3dc606f7ed4c35866176c\": container with ID starting with 6a84c99cd862fccb71661bacec9899ecaeab348650d3dc606f7ed4c35866176c not found: ID does not exist" Nov 03 20:06:35.529556 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:35.529525 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5jhck"] Nov 03 20:06:35.533332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:35.533306 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5jhck"] Nov 03 20:06:35.776487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:35.755560 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26273dd2-22e7-419d-bbc0-cf1f831ef5f8" path="/var/lib/kubelet/pods/26273dd2-22e7-419d-bbc0-cf1f831ef5f8/volumes" Nov 03 20:06:35.811662 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:35.811626 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p7lrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Nov 03 20:06:35.811837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:35.811684 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" podUID="93f7f2c5-96a8-4ccc-ad9b-3cc648f85802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Nov 03 20:06:36.146740 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:06:36.146692 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:06:36.811693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:36.811656 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p7lrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Nov 03 20:06:36.812090 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:36.811723 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" podUID="93f7f2c5-96a8-4ccc-ad9b-3cc648f85802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Nov 03 20:06:37.810966 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:37.810922 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p7lrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Nov 03 20:06:37.811175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:37.810984 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" podUID="93f7f2c5-96a8-4ccc-ad9b-3cc648f85802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Nov 03 20:06:38.290261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:38.290165 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:06:38.815167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:38.815135 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:06:38.816101 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:38.816082 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" Nov 03 20:06:55.667499 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:55.667460 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7"] Nov 03 20:06:55.667911 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:55.667844 4457 cpu_manager.go:414] "containerMap: removing container" podUID="26273dd2-22e7-419d-bbc0-cf1f831ef5f8" containerName="authorino" Nov 03 20:06:55.667911 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:55.667853 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26273dd2-22e7-419d-bbc0-cf1f831ef5f8" containerName="authorino" Nov 03 20:06:55.667989 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:55.667922 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="26273dd2-22e7-419d-bbc0-cf1f831ef5f8" containerName="authorino" Nov 03 20:06:55.670938 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:55.670920 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7" Nov 03 20:06:55.679533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:55.679499 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7"] Nov 03 20:06:55.785235 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:55.785202 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2vpf\" (UniqueName: \"kubernetes.io/projected/428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb-kube-api-access-w2vpf\") pod \"llmisvc-controller-manager-6466c554d8-jz4d7\" (UID: \"428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7" Nov 03 20:06:55.785431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:55.785259 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb-cert\") pod \"llmisvc-controller-manager-6466c554d8-jz4d7\" (UID: \"428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7" Nov 03 20:06:55.886318 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:55.886284 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2vpf\" (UniqueName: \"kubernetes.io/projected/428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb-kube-api-access-w2vpf\") pod \"llmisvc-controller-manager-6466c554d8-jz4d7\" (UID: \"428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7" Nov 03 20:06:55.886553 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:55.886343 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb-cert\") pod \"llmisvc-controller-manager-6466c554d8-jz4d7\" (UID: \"428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7" Nov 03 20:06:55.888788 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:55.888756 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb-cert\") pod \"llmisvc-controller-manager-6466c554d8-jz4d7\" (UID: \"428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7" Nov 03 20:06:55.899282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:55.899258 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2vpf\" (UniqueName: \"kubernetes.io/projected/428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb-kube-api-access-w2vpf\") pod \"llmisvc-controller-manager-6466c554d8-jz4d7\" (UID: \"428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7" Nov 03 20:06:55.985001 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:55.984930 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7" Nov 03 20:06:56.259822 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:56.259797 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7"] Nov 03 20:06:56.261450 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:06:56.261418 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428ef9c3_963d_4e46_b3f4_cf5fcc6cf0cb.slice/crio-bbc87ba12a94c837a26877a1674723176fd95d2384f1ccfa6641be45850f44b8 WatchSource:0}: Error finding container bbc87ba12a94c837a26877a1674723176fd95d2384f1ccfa6641be45850f44b8: Status 404 returned error can't find the container with id bbc87ba12a94c837a26877a1674723176fd95d2384f1ccfa6641be45850f44b8 Nov 03 20:06:56.612684 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:56.612649 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7" event={"ID":"428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb","Type":"ContainerStarted","Data":"bbc87ba12a94c837a26877a1674723176fd95d2384f1ccfa6641be45850f44b8"} Nov 03 20:06:57.618985 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:57.618945 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7" event={"ID":"428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb","Type":"ContainerStarted","Data":"fcf9548089fb6df1565db1bb5289807686ff0b340cdf9a9edba13609eac07bf1"} Nov 03 20:06:57.619366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:57.619028 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7" Nov 03 20:06:57.638863 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:06:57.638820 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7" podStartSLOduration=2.287500068 podStartE2EDuration="2.638805439s" totalImagesPullingTime="351.305371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:06:57.638164742 +0000 UTC m=+596.013761490" watchObservedRunningTime="2026-11-03 20:06:57.638805439 +0000 UTC m=+596.014402186" Nov 03 20:07:01.777816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:01.777782 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:07:01.778959 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:01.778928 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:07:01.792812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:01.792793 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:07:01.793782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:01.793762 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:07:06.144644 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:07:06.144605 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:07:28.625921 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:28.625886 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-jz4d7" Nov 03 20:07:28.674474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:28.674446 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c"] Nov 03 20:07:28.674706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:28.674678 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" podUID="ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f" containerName="manager" containerID="cri-o://06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4" gracePeriod=30 Nov 03 20:07:28.726801 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:07:28.726769 4457 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-poded8f5019_f8d7_40ab_b31a_cd36b9f9b67f.slice/crio-conmon-06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:07:28.921894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:28.921854 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" Nov 03 20:07:28.962599 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:28.962568 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-kube-api-access-52zxh\" (UniqueName: \"kubernetes.io/projected/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-kube-api-access-52zxh\") pod \"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f\" (UID: \"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f\") " Nov 03 20:07:28.962749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:28.962641 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-cert\" (UniqueName: \"kubernetes.io/secret/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-cert\") pod \"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f\" (UID: \"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f\") " Nov 03 20:07:28.964659 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:28.964629 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-cert" pod "ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f" (UID: "ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:07:28.964765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:28.964738 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-kube-api-access-52zxh" pod "ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f" (UID: "ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f"). InnerVolumeSpecName "kube-api-access-52zxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:07:29.063933 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:29.063898 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52zxh\" (UniqueName: \"kubernetes.io/projected/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-kube-api-access-52zxh\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:29.063933 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:29.063927 4457 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:07:29.779753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:29.779720 4457 generic.go:356] "Generic (PLEG): container finished" podID="ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f" containerID="06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4" exitCode=0 Nov 03 20:07:29.780200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:29.779762 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" event={"ID":"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f","Type":"ContainerDied","Data":"06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4"} Nov 03 20:07:29.780200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:29.779785 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" Nov 03 20:07:29.780200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:29.779822 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c" event={"ID":"ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f","Type":"ContainerDied","Data":"12644fed7e587c81cd52a33986a6ddf7b1e5e6dcd21a38bce82e6e6400e40032"} Nov 03 20:07:29.780200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:29.779844 4457 scope.go:122] "RemoveContainer" containerID="06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4" Nov 03 20:07:29.788578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:29.788558 4457 scope.go:122] "RemoveContainer" containerID="06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4" Nov 03 20:07:29.788830 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:07:29.788814 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4\": container with ID starting with 06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4 not found: ID does not exist" containerID="06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4" Nov 03 20:07:29.788872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:29.788838 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4"} err="failed to get container status \"06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4\": rpc error: code = NotFound desc = could not find container \"06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4\": container with ID starting with 06dd7994d8690cc9063234cccdcccb75fd1ea92d1416dcd2f1ffc331ec0d47c4 not found: ID does not exist" Nov 03 20:07:29.799287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:29.799259 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c"] Nov 03 20:07:29.810189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:29.810163 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-5bk7c"] Nov 03 20:07:31.755305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:07:31.755268 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f" path="/var/lib/kubelet/pods/ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f/volumes" Nov 03 20:07:36.144375 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:07:36.144336 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:08:06.144131 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:08:06.144087 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:08:19.046919 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:19.046880 4457 generic.go:356] "Generic (PLEG): container finished" podID="2bb0a4bf-7a65-40ea-9a42-6d1d89c76645" containerID="4cacf177178e5818c08b46ac32f56a2bf1c417ae930a3fa3388b54abd1e4dc95" exitCode=1 Nov 03 20:08:19.047487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:19.046926 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" event={"ID":"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645","Type":"ContainerDied","Data":"4cacf177178e5818c08b46ac32f56a2bf1c417ae930a3fa3388b54abd1e4dc95"} Nov 03 20:08:19.047487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:19.047338 4457 scope.go:122] "RemoveContainer" containerID="4cacf177178e5818c08b46ac32f56a2bf1c417ae930a3fa3388b54abd1e4dc95" Nov 03 20:08:19.193296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:19.193218 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:08:20.053572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:20.053534 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" event={"ID":"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645","Type":"ContainerStarted","Data":"24ba90871525bf113d450bcb9f142c5c0c55f045bca1fe3ea872a4138860370c"} Nov 03 20:08:20.053992 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:20.053780 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:08:31.061325 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:31.061289 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:08:36.144648 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:08:36.144607 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:08:39.904937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.904903 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m"] Nov 03 20:08:39.905477 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.905301 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f" containerName="manager" Nov 03 20:08:39.905477 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.905313 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f" containerName="manager" Nov 03 20:08:39.905477 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.905396 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8f5019-f8d7-40ab-b31a-cd36b9f9b67f" containerName="manager" Nov 03 20:08:39.910286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.910253 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:39.924420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.924376 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m"] Nov 03 20:08:39.937448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.937420 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp"] Nov 03 20:08:39.942530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.942504 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:39.957996 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.957967 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp"] Nov 03 20:08:39.987337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987297 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:39.987337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987346 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:39.987611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987458 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:39.987611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987500 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:39.987611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987523 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:39.987611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987557 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:39.987611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987573 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c6aa2c-4423-4872-85e3-33983cb81c2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:39.987611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987609 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qd7\" (UniqueName: \"kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-kube-api-access-l7qd7\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:39.987923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987633 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:39.987923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987652 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:39.987923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987666 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:39.987923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987687 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:39.987923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987705 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:39.987923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987727 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9s2n\" (UniqueName: \"kubernetes.io/projected/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-kube-api-access-d9s2n\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:39.987923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987750 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:39.987923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987778 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:39.987923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987796 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:39.987923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:39.987824 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.089234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089198 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.089503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089240 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.089503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089258 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.089503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089290 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.089503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089321 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.089503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089350 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.089503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089375 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.089503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089435 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c6aa2c-4423-4872-85e3-33983cb81c2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.089503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089485 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qd7\" (UniqueName: \"kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-kube-api-access-l7qd7\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.089988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089523 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.089988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089548 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.089988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089573 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.089988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089606 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.089988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089633 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.089988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089668 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9s2n\" (UniqueName: \"kubernetes.io/projected/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-kube-api-access-d9s2n\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.089988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089705 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.089988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089748 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.089988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089777 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.089988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089806 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.089988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.089869 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.090569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.090121 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.090569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.090131 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.090569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.090199 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.090569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.090429 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.090569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.090478 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.090828 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.090718 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.091060 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.091035 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.091181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.091158 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c6aa2c-4423-4872-85e3-33983cb81c2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.092326 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.092302 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.092920 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.092903 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.093004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.092921 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.093306 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.093286 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.101504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.101478 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.101862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.101837 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9s2n\" (UniqueName: \"kubernetes.io/projected/8f7c5959-df8b-4348-8c8c-c61ba06f66d4-kube-api-access-d9s2n\") pod \"maas-default-gateway-openshift-default-77f556d647-dgjgp\" (UID: \"8f7c5959-df8b-4348-8c8c-c61ba06f66d4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.102434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.102412 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.103197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.103173 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qd7\" (UniqueName: \"kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-kube-api-access-l7qd7\") pod \"maas-default-gateway-openshift-default-77f556d647-5bz7m\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.229891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.229791 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:40.257493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.257466 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:40.527352 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.527322 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m"] Nov 03 20:08:40.529314 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:08:40.529283 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c6aa2c_4423_4872_85e3_33983cb81c2b.slice/crio-e0f9364ff310395620efadedd52b959228235172fbb35e12c1adb46193b10564 WatchSource:0}: Error finding container e0f9364ff310395620efadedd52b959228235172fbb35e12c1adb46193b10564: Status 404 returned error can't find the container with id e0f9364ff310395620efadedd52b959228235172fbb35e12c1adb46193b10564 Nov 03 20:08:40.532444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.532380 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:40.532527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.532496 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:40.532575 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.532546 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:40.562022 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.561982 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp"] Nov 03 20:08:40.563584 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:08:40.563553 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7c5959_df8b_4348_8c8c_c61ba06f66d4.slice/crio-bc43ed3d1e7e74e05af7195f9a9bc09abd2553d88401139c00947abe8f91c397 WatchSource:0}: Error finding container bc43ed3d1e7e74e05af7195f9a9bc09abd2553d88401139c00947abe8f91c397: Status 404 returned error can't find the container with id bc43ed3d1e7e74e05af7195f9a9bc09abd2553d88401139c00947abe8f91c397 Nov 03 20:08:40.565943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.565910 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:40.566049 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.565979 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:40.566049 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:40.566009 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:08:41.180417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.179795 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" event={"ID":"18c6aa2c-4423-4872-85e3-33983cb81c2b","Type":"ContainerStarted","Data":"eefdb561dfcd553f17780b48e96c91fc94ba17d822776fc152ff06808d24f714"} Nov 03 20:08:41.180417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.179863 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" event={"ID":"18c6aa2c-4423-4872-85e3-33983cb81c2b","Type":"ContainerStarted","Data":"e0f9364ff310395620efadedd52b959228235172fbb35e12c1adb46193b10564"} Nov 03 20:08:41.183990 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.183318 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" event={"ID":"8f7c5959-df8b-4348-8c8c-c61ba06f66d4","Type":"ContainerStarted","Data":"1bd23ca1eb578f4f8da4bea25569fb33ac6aa8f4551db59a2e6a984844d5a476"} Nov 03 20:08:41.183990 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.183375 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" event={"ID":"8f7c5959-df8b-4348-8c8c-c61ba06f66d4","Type":"ContainerStarted","Data":"bc43ed3d1e7e74e05af7195f9a9bc09abd2553d88401139c00947abe8f91c397"} Nov 03 20:08:41.231410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.224264 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79b6656fd9-m22gh"] Nov 03 20:08:41.248888 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.245243 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-79b6656fd9-m22gh" podUID="7e4e5eed-61cb-4444-b0f9-a398d67ab773" containerName="authorino" containerID="cri-o://95dd97a6669086a8da2af5279fcb6f047d3cc297d9d4e7b3415372db608b008b" gracePeriod=30 Nov 03 20:08:41.249076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.245953 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:41.249076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.248772 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5bz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:08:41.249218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.249078 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 20:08:41.290008 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.289945 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" podStartSLOduration=2.289925597 podStartE2EDuration="2.289925597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:41.248369535 +0000 UTC m=+699.623966290" watchObservedRunningTime="2026-11-03 20:08:41.289925597 +0000 UTC m=+699.665522345" Nov 03 20:08:41.291371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.291252 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:41.328011 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.327975 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:08:41.329324 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.329287 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" podUID="8f7c5959-df8b-4348-8c8c-c61ba06f66d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 20:08:41.353950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:41.353887 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" podStartSLOduration=2.353858473 podStartE2EDuration="2.353858473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:08:41.352613099 +0000 UTC m=+699.728209851" watchObservedRunningTime="2026-11-03 20:08:41.353858473 +0000 UTC m=+699.729455219" Nov 03 20:08:42.166403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.166356 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b6656fd9-m22gh" Nov 03 20:08:42.238441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.234695 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5bz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:08:42.258265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.258149 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 20:08:42.327686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.327003 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4e5eed-61cb-4444-b0f9-a398d67ab773-kube-api-access-bpf7t\" (UniqueName: \"kubernetes.io/projected/7e4e5eed-61cb-4444-b0f9-a398d67ab773-kube-api-access-bpf7t\") pod \"7e4e5eed-61cb-4444-b0f9-a398d67ab773\" (UID: \"7e4e5eed-61cb-4444-b0f9-a398d67ab773\") " Nov 03 20:08:42.327686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.327112 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e4e5eed-61cb-4444-b0f9-a398d67ab773-tls-cert\" (UniqueName: \"kubernetes.io/secret/7e4e5eed-61cb-4444-b0f9-a398d67ab773-tls-cert\") pod \"7e4e5eed-61cb-4444-b0f9-a398d67ab773\" (UID: \"7e4e5eed-61cb-4444-b0f9-a398d67ab773\") " Nov 03 20:08:42.345791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.345760 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:08:42.353356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.352574 4457 generic.go:356] "Generic (PLEG): container finished" podID="7e4e5eed-61cb-4444-b0f9-a398d67ab773" containerID="95dd97a6669086a8da2af5279fcb6f047d3cc297d9d4e7b3415372db608b008b" exitCode=0 Nov 03 20:08:42.353356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.352594 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" podUID="8f7c5959-df8b-4348-8c8c-c61ba06f66d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 20:08:42.353356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.352688 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-79b6656fd9-m22gh" Nov 03 20:08:42.353356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.352851 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79b6656fd9-m22gh" event={"ID":"7e4e5eed-61cb-4444-b0f9-a398d67ab773","Type":"ContainerDied","Data":"95dd97a6669086a8da2af5279fcb6f047d3cc297d9d4e7b3415372db608b008b"} Nov 03 20:08:42.353356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.352883 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-79b6656fd9-m22gh" event={"ID":"7e4e5eed-61cb-4444-b0f9-a398d67ab773","Type":"ContainerDied","Data":"ead42e8404018ebfb5a36521b2b04dd8cc04bb749bfc70d5eb33380c21f8c661"} Nov 03 20:08:42.353356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.352903 4457 scope.go:122] "RemoveContainer" containerID="95dd97a6669086a8da2af5279fcb6f047d3cc297d9d4e7b3415372db608b008b" Nov 03 20:08:42.444365 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.444315 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4e5eed-61cb-4444-b0f9-a398d67ab773-kube-api-access-bpf7t" pod "7e4e5eed-61cb-4444-b0f9-a398d67ab773" (UID: "7e4e5eed-61cb-4444-b0f9-a398d67ab773"). InnerVolumeSpecName "kube-api-access-bpf7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:08:42.454189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.454139 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e4e5eed-61cb-4444-b0f9-a398d67ab773-tls-cert" pod "7e4e5eed-61cb-4444-b0f9-a398d67ab773" (UID: "7e4e5eed-61cb-4444-b0f9-a398d67ab773"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:08:42.500064 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.499874 4457 scope.go:122] "RemoveContainer" containerID="95dd97a6669086a8da2af5279fcb6f047d3cc297d9d4e7b3415372db608b008b" Nov 03 20:08:42.503926 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:08:42.503804 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95dd97a6669086a8da2af5279fcb6f047d3cc297d9d4e7b3415372db608b008b\": container with ID starting with 95dd97a6669086a8da2af5279fcb6f047d3cc297d9d4e7b3415372db608b008b not found: ID does not exist" containerID="95dd97a6669086a8da2af5279fcb6f047d3cc297d9d4e7b3415372db608b008b" Nov 03 20:08:42.503926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.503848 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95dd97a6669086a8da2af5279fcb6f047d3cc297d9d4e7b3415372db608b008b"} err="failed to get container status \"95dd97a6669086a8da2af5279fcb6f047d3cc297d9d4e7b3415372db608b008b\": rpc error: code = NotFound desc = could not find container \"95dd97a6669086a8da2af5279fcb6f047d3cc297d9d4e7b3415372db608b008b\": container with ID starting with 95dd97a6669086a8da2af5279fcb6f047d3cc297d9d4e7b3415372db608b008b not found: ID does not exist" Nov 03 20:08:42.530074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.529948 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpf7t\" (UniqueName: \"kubernetes.io/projected/7e4e5eed-61cb-4444-b0f9-a398d67ab773-kube-api-access-bpf7t\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.530074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.529991 4457 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7e4e5eed-61cb-4444-b0f9-a398d67ab773-tls-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:08:42.735424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.733265 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-79b6656fd9-m22gh"] Nov 03 20:08:42.744107 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:42.744079 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-79b6656fd9-m22gh"] Nov 03 20:08:43.230651 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:43.230610 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5bz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:08:43.230831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:43.230673 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 20:08:43.259769 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:43.259735 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:08:43.260307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:43.260276 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" podUID="8f7c5959-df8b-4348-8c8c-c61ba06f66d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 20:08:43.760494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:43.760448 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4e5eed-61cb-4444-b0f9-a398d67ab773" path="/var/lib/kubelet/pods/7e4e5eed-61cb-4444-b0f9-a398d67ab773/volumes" Nov 03 20:08:44.230343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:44.230307 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5bz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:08:44.230551 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:44.230357 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 20:08:44.262412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:44.259722 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:08:44.262412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:44.259778 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" podUID="8f7c5959-df8b-4348-8c8c-c61ba06f66d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 20:08:45.230763 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:45.230719 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5bz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:08:45.230941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:45.230770 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 20:08:45.259778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:45.259740 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:08:45.260044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:45.260016 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" podUID="8f7c5959-df8b-4348-8c8c-c61ba06f66d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 20:08:46.230655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:46.230621 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5bz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:08:46.231062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:46.230670 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 20:08:46.261410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:46.259715 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:08:46.261410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:46.259772 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" podUID="8f7c5959-df8b-4348-8c8c-c61ba06f66d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 20:08:47.231194 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:47.231117 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5bz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:08:47.232231 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:47.232200 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 20:08:47.258755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:47.258338 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:08:47.258755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:47.258423 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" podUID="8f7c5959-df8b-4348-8c8c-c61ba06f66d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 20:08:48.230372 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:48.230326 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5bz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Nov 03 20:08:48.230592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:48.230413 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Nov 03 20:08:48.258448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:48.258407 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 03 20:08:48.258856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:48.258477 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" podUID="8f7c5959-df8b-4348-8c8c-c61ba06f66d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 03 20:08:50.272648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:50.272597 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5bz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:08:50.273302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:50.273265 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:50.299588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:50.299545 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:08:50.299828 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:50.299793 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" podUID="8f7c5959-df8b-4348-8c8c-c61ba06f66d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:51.315570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:51.315528 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5bz7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:08:51.316099 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:51.315598 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:51.341514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:51.341484 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dgjgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:08:51.341632 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:51.341530 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" podUID="8f7c5959-df8b-4348-8c8c-c61ba06f66d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:08:51.787288 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:51.787192 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:51.787622 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:51.787589 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:51.788414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:51.788373 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:08:51.828576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:51.828535 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:51.829068 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:51.829045 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:08:51.829836 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:08:51.829816 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" Nov 03 20:09:06.144106 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:09:06.144070 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:09:36.144997 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:09:36.144959 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:10:00.669988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.669950 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f"] Nov 03 20:10:00.671036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.671006 4457 cpu_manager.go:414] "containerMap: removing container" podUID="7e4e5eed-61cb-4444-b0f9-a398d67ab773" containerName="authorino" Nov 03 20:10:00.671036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.671030 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4e5eed-61cb-4444-b0f9-a398d67ab773" containerName="authorino" Nov 03 20:10:00.671205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.671138 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4e5eed-61cb-4444-b0f9-a398d67ab773" containerName="authorino" Nov 03 20:10:00.674605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.674584 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.687102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.687070 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f"] Nov 03 20:10:00.687242 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.687196 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-b1b190-openshift-default-dockercfg-g8zqz\"" Nov 03 20:10:00.719954 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.719893 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.720226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.720197 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.720419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.720254 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942423a8-2c01-4ef8-82df-cf8892de3585-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.720419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.720308 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942423a8-2c01-4ef8-82df-cf8892de3585-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.720419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.720342 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.720419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.720381 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.720652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.720455 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.720652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.720509 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcx9n\" (UniqueName: \"kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-kube-api-access-lcx9n\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.720756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.720706 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.757412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.756710 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf"] Nov 03 20:10:00.764909 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.764881 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.774190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.774159 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf"] Nov 03 20:10:00.780565 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.780520 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-90ef9c-openshift-default-dockercfg-9s9f5\"" Nov 03 20:10:00.822007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.821970 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.822007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822010 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.822254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822032 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cbfb85-a563-4265-828a-40ead144734b-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.822254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822056 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.822254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822121 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.822254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822166 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.822254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822189 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.822254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822226 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcx9n\" (UniqueName: \"kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-kube-api-access-lcx9n\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.822254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822246 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.822637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822270 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qk99\" (UniqueName: \"kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-kube-api-access-7qk99\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.822637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822288 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.822637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822321 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.822637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822345 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.822637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822355 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.822637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822379 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.822637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822423 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942423a8-2c01-4ef8-82df-cf8892de3585-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.822637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822461 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.822637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822466 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942423a8-2c01-4ef8-82df-cf8892de3585-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.822637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822521 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.822637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.822556 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cbfb85-a563-4265-828a-40ead144734b-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.823276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.823252 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942423a8-2c01-4ef8-82df-cf8892de3585-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.823565 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.823541 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.823675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.823645 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.825557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.825533 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942423a8-2c01-4ef8-82df-cf8892de3585-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.826371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.826348 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.841399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.841359 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcx9n\" (UniqueName: \"kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-kube-api-access-lcx9n\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.841592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.841488 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:00.923964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.923866 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cbfb85-a563-4265-828a-40ead144734b-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.923964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.923911 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.923964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.923937 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.924237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.923969 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.924237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.924009 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.924237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.924042 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qk99\" (UniqueName: \"kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-kube-api-access-7qk99\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.924237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.924082 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.924237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.924174 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.924237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.924205 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cbfb85-a563-4265-828a-40ead144734b-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.924586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.924350 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.924586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.924415 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.924708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.924648 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.924766 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.924711 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.924948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.924926 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cbfb85-a563-4265-828a-40ead144734b-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.926491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.926469 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.927075 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.927050 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cbfb85-a563-4265-828a-40ead144734b-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.941795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.941765 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qk99\" (UniqueName: \"kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-kube-api-access-7qk99\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:00.942067 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:00.942046 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:01.000914 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.000873 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:01.085289 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.084977 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:01.337614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.337589 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f"] Nov 03 20:10:01.338780 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:10:01.338753 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942423a8_2c01_4ef8_82df_cf8892de3585.slice/crio-15ad228c1c72705939d5fbde867b76d995b3f0a927969e9bfc06505fffdf0331 WatchSource:0}: Error finding container 15ad228c1c72705939d5fbde867b76d995b3f0a927969e9bfc06505fffdf0331: Status 404 returned error can't find the container with id 15ad228c1c72705939d5fbde867b76d995b3f0a927969e9bfc06505fffdf0331 Nov 03 20:10:01.341516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.341485 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:01.341624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.341564 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:01.341624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.341611 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:01.409090 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.407275 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf"] Nov 03 20:10:01.410594 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:10:01.410561 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cbfb85_a563_4265_828a_40ead144734b.slice/crio-860518f7694a8f2bfeed25639e79266b813934ad90dcee1049f830fe69776ba2 WatchSource:0}: Error finding container 860518f7694a8f2bfeed25639e79266b813934ad90dcee1049f830fe69776ba2: Status 404 returned error can't find the container with id 860518f7694a8f2bfeed25639e79266b813934ad90dcee1049f830fe69776ba2 Nov 03 20:10:01.413146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.413109 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:01.413249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.413179 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:01.413249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.413208 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:01.825346 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.825299 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" event={"ID":"49cbfb85-a563-4265-828a-40ead144734b","Type":"ContainerStarted","Data":"fa891d3b4bad4521461ace4728816302f86a2b8ece0f8d5e701917cc39c88280"} Nov 03 20:10:01.825918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.825379 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" event={"ID":"49cbfb85-a563-4265-828a-40ead144734b","Type":"ContainerStarted","Data":"860518f7694a8f2bfeed25639e79266b813934ad90dcee1049f830fe69776ba2"} Nov 03 20:10:01.827151 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.827125 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" event={"ID":"942423a8-2c01-4ef8-82df-cf8892de3585","Type":"ContainerStarted","Data":"bd1119a197ada5445a2ce8cdb33836c21d6ee8ab27178de9b21ead66762627cb"} Nov 03 20:10:01.827298 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.827168 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" event={"ID":"942423a8-2c01-4ef8-82df-cf8892de3585","Type":"ContainerStarted","Data":"15ad228c1c72705939d5fbde867b76d995b3f0a927969e9bfc06505fffdf0331"} Nov 03 20:10:01.850926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.850870 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" podStartSLOduration=1.85085153 podStartE2EDuration="1.85085153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:01.849063666 +0000 UTC m=+780.224660415" watchObservedRunningTime="2026-11-03 20:10:01.85085153 +0000 UTC m=+780.226448277" Nov 03 20:10:01.874883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:01.874823 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" podStartSLOduration=1.874802955 podStartE2EDuration="1.874802955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:01.873098958 +0000 UTC m=+780.248695719" watchObservedRunningTime="2026-11-03 20:10:01.874802955 +0000 UTC m=+780.250399704" Nov 03 20:10:02.009417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:02.009368 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:02.018578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:02.018130 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:02.085732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:02.085640 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:02.092217 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:02.092188 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:02.832302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:02.832272 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:02.832791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:02.832326 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:02.833155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:02.833130 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:02.833310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:02.833292 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:06.150564 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:10:06.150522 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:10:06.744659 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:06.744623 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4"] Nov 03 20:10:06.748639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:06.748615 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" Nov 03 20:10:06.756150 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:06.756125 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b1b190\"" Nov 03 20:10:06.759729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:06.759705 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4"] Nov 03 20:10:06.796369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:06.796320 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c875b5-372c-42fe-828a-7605988b595d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4\" (UID: \"54c875b5-372c-42fe-828a-7605988b595d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" Nov 03 20:10:06.796586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:06.796479 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqd87\" (UniqueName: \"kubernetes.io/projected/54c875b5-372c-42fe-828a-7605988b595d-kube-api-access-zqd87\") pod \"payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4\" (UID: \"54c875b5-372c-42fe-828a-7605988b595d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" Nov 03 20:10:06.805071 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:06.805010 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b1b190-dockercfg-c7jm7\"" Nov 03 20:10:06.901412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:06.897955 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c875b5-372c-42fe-828a-7605988b595d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4\" (UID: \"54c875b5-372c-42fe-828a-7605988b595d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" Nov 03 20:10:06.901412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:06.898038 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqd87\" (UniqueName: \"kubernetes.io/projected/54c875b5-372c-42fe-828a-7605988b595d-kube-api-access-zqd87\") pod \"payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4\" (UID: \"54c875b5-372c-42fe-828a-7605988b595d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" Nov 03 20:10:06.901412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:06.899424 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c875b5-372c-42fe-828a-7605988b595d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4\" (UID: \"54c875b5-372c-42fe-828a-7605988b595d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" Nov 03 20:10:06.916950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:06.916907 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqd87\" (UniqueName: \"kubernetes.io/projected/54c875b5-372c-42fe-828a-7605988b595d-kube-api-access-zqd87\") pod \"payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4\" (UID: \"54c875b5-372c-42fe-828a-7605988b595d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" Nov 03 20:10:07.065589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.065554 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" Nov 03 20:10:07.357257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.357230 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4"] Nov 03 20:10:07.362743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.362721 4457 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:10:07.445416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.445359 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg"] Nov 03 20:10:07.450652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.450630 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" Nov 03 20:10:07.464416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.464291 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-8013d9-dockercfg-8zbj9\"" Nov 03 20:10:07.467233 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.467193 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg"] Nov 03 20:10:07.503500 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.503470 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc0361e-b362-496f-8d29-ac11c012caa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg\" (UID: \"3bc0361e-b362-496f-8d29-ac11c012caa6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" Nov 03 20:10:07.503669 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.503527 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtfkm\" (UniqueName: \"kubernetes.io/projected/3bc0361e-b362-496f-8d29-ac11c012caa6-kube-api-access-qtfkm\") pod \"payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg\" (UID: \"3bc0361e-b362-496f-8d29-ac11c012caa6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" Nov 03 20:10:07.527153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.527126 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-8013d9\"" Nov 03 20:10:07.604884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.604844 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc0361e-b362-496f-8d29-ac11c012caa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg\" (UID: \"3bc0361e-b362-496f-8d29-ac11c012caa6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" Nov 03 20:10:07.605094 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.604953 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtfkm\" (UniqueName: \"kubernetes.io/projected/3bc0361e-b362-496f-8d29-ac11c012caa6-kube-api-access-qtfkm\") pod \"payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg\" (UID: \"3bc0361e-b362-496f-8d29-ac11c012caa6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" Nov 03 20:10:07.605738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.605703 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc0361e-b362-496f-8d29-ac11c012caa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg\" (UID: \"3bc0361e-b362-496f-8d29-ac11c012caa6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" Nov 03 20:10:07.617953 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.617880 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtfkm\" (UniqueName: \"kubernetes.io/projected/3bc0361e-b362-496f-8d29-ac11c012caa6-kube-api-access-qtfkm\") pod \"payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg\" (UID: \"3bc0361e-b362-496f-8d29-ac11c012caa6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" Nov 03 20:10:07.771486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.771451 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" Nov 03 20:10:07.919867 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:07.919783 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" event={"ID":"54c875b5-372c-42fe-828a-7605988b595d","Type":"ContainerStarted","Data":"24089de8658e281f5755ffdfe713fc1e17b519d7629f385419e3d7f73639ec17"} Nov 03 20:10:08.088974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:08.088938 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg"] Nov 03 20:10:08.090380 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:10:08.090347 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc0361e_b362_496f_8d29_ac11c012caa6.slice/crio-06a1d5ad61c6892ca0c1a3a4e4bb376881d9afb2403bf2de7901db963b5f1591 WatchSource:0}: Error finding container 06a1d5ad61c6892ca0c1a3a4e4bb376881d9afb2403bf2de7901db963b5f1591: Status 404 returned error can't find the container with id 06a1d5ad61c6892ca0c1a3a4e4bb376881d9afb2403bf2de7901db963b5f1591 Nov 03 20:10:08.936433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:08.934943 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" event={"ID":"3bc0361e-b362-496f-8d29-ac11c012caa6","Type":"ContainerStarted","Data":"06a1d5ad61c6892ca0c1a3a4e4bb376881d9afb2403bf2de7901db963b5f1591"} Nov 03 20:10:09.231891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.231799 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt"] Nov 03 20:10:09.241053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.238251 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:09.241053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.240984 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt"] Nov 03 20:10:09.249970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.249343 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-b1b190-serving-cert\"" Nov 03 20:10:09.249970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.249498 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:10:09.249970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.249635 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-znhr8\"" Nov 03 20:10:09.323081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.323037 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5hh6\" (UniqueName: \"kubernetes.io/projected/0149807d-7ed5-4e99-a457-04532e25b0a7-kube-api-access-n5hh6\") pod \"maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:09.323262 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.323141 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:09.323262 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.323170 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-metrics-certs\") pod \"maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:09.323412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.323363 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149807d-7ed5-4e99-a457-04532e25b0a7-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:09.425019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.424859 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149807d-7ed5-4e99-a457-04532e25b0a7-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:09.425019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.424940 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5hh6\" (UniqueName: \"kubernetes.io/projected/0149807d-7ed5-4e99-a457-04532e25b0a7-kube-api-access-n5hh6\") pod \"maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:09.425019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.425003 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:09.425019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.425037 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-metrics-certs\") pod \"maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:09.425689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.425584 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149807d-7ed5-4e99-a457-04532e25b0a7-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:09.425689 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:10:09.425671 4457 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-serving-cert: secret "maas-api-e2e-worker-w0-b1b190-serving-cert" not found Nov 03 20:10:09.425776 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:10:09.425725 4457 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-maas-api-tls podName:0149807d-7ed5-4e99-a457-04532e25b0a7 nodeName:}" failed. No retries permitted until 2026-11-03 20:10:09.925707411 +0000 UTC m=+788.301304153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-maas-api-tls") pod "maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" (UID: "0149807d-7ed5-4e99-a457-04532e25b0a7") : secret "maas-api-e2e-worker-w0-b1b190-serving-cert" not found Nov 03 20:10:09.428250 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.428093 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-metrics-certs\") pod \"maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:09.440574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.440542 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5hh6\" (UniqueName: \"kubernetes.io/projected/0149807d-7ed5-4e99-a457-04532e25b0a7-kube-api-access-n5hh6\") pod \"maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:09.931349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.931306 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:09.934101 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:09.934056 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:10.170593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:10.170556 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:10.554827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:10.554779 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt"] Nov 03 20:10:10.949519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:10.949440 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" event={"ID":"54c875b5-372c-42fe-828a-7605988b595d","Type":"ContainerStarted","Data":"fe80e7cabbbff7b9e436b7e98a7b620a4b1da0df852dfb8836c8fb7c09258cfb"} Nov 03 20:10:10.949658 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:10.949521 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" Nov 03 20:10:10.950843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:10.950822 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" event={"ID":"0149807d-7ed5-4e99-a457-04532e25b0a7","Type":"ContainerStarted","Data":"c295b80d6ca3b7ed545090a18c38e91d4b7f0dd86b3a17c3ed91b68b4de3e511"} Nov 03 20:10:11.960183 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:11.960145 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" event={"ID":"3bc0361e-b362-496f-8d29-ac11c012caa6","Type":"ContainerStarted","Data":"08cc367d3a242cee5e47df3ee9479bb4de4687409b6d7132527e4061874dbd95"} Nov 03 20:10:11.960596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:11.960408 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" Nov 03 20:10:11.979575 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:11.979516 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" podStartSLOduration=2.6894267640000002 podStartE2EDuration="5.979499942s" totalImagesPullingTime="3.290073178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:10.968989482 +0000 UTC m=+789.344586230" watchObservedRunningTime="2026-11-03 20:10:11.979499942 +0000 UTC m=+790.355096688" Nov 03 20:10:11.979873 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:11.979849 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" podStartSLOduration=2.119255264 podStartE2EDuration="4.97984162s" totalImagesPullingTime="2.860586356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:11.978375123 +0000 UTC m=+790.353971872" watchObservedRunningTime="2026-11-03 20:10:11.97984162 +0000 UTC m=+790.355438366" Nov 03 20:10:12.968628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:12.968484 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" event={"ID":"0149807d-7ed5-4e99-a457-04532e25b0a7","Type":"ContainerStarted","Data":"d37d76de3b09c5c7f0378319ea654cfd175593ce3227e0b7226910b42a804369"} Nov 03 20:10:12.991177 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:12.991079 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" podStartSLOduration=1.9304645489999999 podStartE2EDuration="3.991060156s" totalImagesPullingTime="2.060595607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:12.988968815 +0000 UTC m=+791.364565574" watchObservedRunningTime="2026-11-03 20:10:12.991060156 +0000 UTC m=+791.366656905" Nov 03 20:10:13.987500 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:13.987451 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:15.388661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.388621 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l"] Nov 03 20:10:15.392640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.392618 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" Nov 03 20:10:15.401497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.400469 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-3aba31\"" Nov 03 20:10:15.405415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.405378 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l"] Nov 03 20:10:15.406058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.406038 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de4b055-a669-4760-ba03-041291c506fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l\" (UID: \"4de4b055-a669-4760-ba03-041291c506fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" Nov 03 20:10:15.406205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.406099 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhch\" (UniqueName: \"kubernetes.io/projected/4de4b055-a669-4760-ba03-041291c506fd-kube-api-access-vdhch\") pod \"payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l\" (UID: \"4de4b055-a669-4760-ba03-041291c506fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" Nov 03 20:10:15.415340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.415319 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g"] Nov 03 20:10:15.418833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.418816 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" Nov 03 20:10:15.427751 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.427729 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-3aba31-dockercfg-r8tnc\"" Nov 03 20:10:15.428159 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.428135 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g"] Nov 03 20:10:15.507048 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.507007 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhch\" (UniqueName: \"kubernetes.io/projected/4de4b055-a669-4760-ba03-041291c506fd-kube-api-access-vdhch\") pod \"payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l\" (UID: \"4de4b055-a669-4760-ba03-041291c506fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" Nov 03 20:10:15.507319 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.507290 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de4b055-a669-4760-ba03-041291c506fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l\" (UID: \"4de4b055-a669-4760-ba03-041291c506fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" Nov 03 20:10:15.508282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.508258 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de4b055-a669-4760-ba03-041291c506fd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l\" (UID: \"4de4b055-a669-4760-ba03-041291c506fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" Nov 03 20:10:15.520315 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.520289 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhch\" (UniqueName: \"kubernetes.io/projected/4de4b055-a669-4760-ba03-041291c506fd-kube-api-access-vdhch\") pod \"payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l\" (UID: \"4de4b055-a669-4760-ba03-041291c506fd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" Nov 03 20:10:15.608258 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.608216 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250b873d-632c-4169-b357-77f6d8b9e9b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g\" (UID: \"250b873d-632c-4169-b357-77f6d8b9e9b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" Nov 03 20:10:15.608453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.608353 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv25h\" (UniqueName: \"kubernetes.io/projected/250b873d-632c-4169-b357-77f6d8b9e9b6-kube-api-access-bv25h\") pod \"payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g\" (UID: \"250b873d-632c-4169-b357-77f6d8b9e9b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" Nov 03 20:10:15.707051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.706968 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" Nov 03 20:10:15.709447 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.709417 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250b873d-632c-4169-b357-77f6d8b9e9b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g\" (UID: \"250b873d-632c-4169-b357-77f6d8b9e9b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" Nov 03 20:10:15.709579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.709561 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv25h\" (UniqueName: \"kubernetes.io/projected/250b873d-632c-4169-b357-77f6d8b9e9b6-kube-api-access-bv25h\") pod \"payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g\" (UID: \"250b873d-632c-4169-b357-77f6d8b9e9b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" Nov 03 20:10:15.710346 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.710294 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250b873d-632c-4169-b357-77f6d8b9e9b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g\" (UID: \"250b873d-632c-4169-b357-77f6d8b9e9b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" Nov 03 20:10:15.726075 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.726046 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv25h\" (UniqueName: \"kubernetes.io/projected/250b873d-632c-4169-b357-77f6d8b9e9b6-kube-api-access-bv25h\") pod \"payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g\" (UID: \"250b873d-632c-4169-b357-77f6d8b9e9b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" Nov 03 20:10:15.739885 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:15.739864 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" Nov 03 20:10:16.045171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.045145 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l"] Nov 03 20:10:16.094853 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.094817 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g"] Nov 03 20:10:16.095941 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:10:16.095914 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250b873d_632c_4169_b357_77f6d8b9e9b6.slice/crio-627a83be6254c7676c30a4329aaf9075c91de4264af8017f393034259032651e WatchSource:0}: Error finding container 627a83be6254c7676c30a4329aaf9075c91de4264af8017f393034259032651e: Status 404 returned error can't find the container with id 627a83be6254c7676c30a4329aaf9075c91de4264af8017f393034259032651e Nov 03 20:10:16.213428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.208823 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t"] Nov 03 20:10:16.219738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.219707 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t"] Nov 03 20:10:16.219965 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.219818 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" Nov 03 20:10:16.260466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.255478 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-62221d\"" Nov 03 20:10:16.275424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.260784 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-62221d-dockercfg-hndkc\"" Nov 03 20:10:16.426592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.426559 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t\" (UID: \"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" Nov 03 20:10:16.426592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.426604 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k22hp\" (UniqueName: \"kubernetes.io/projected/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-kube-api-access-k22hp\") pod \"payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t\" (UID: \"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" Nov 03 20:10:16.527690 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.527640 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t\" (UID: \"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" Nov 03 20:10:16.527872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.527755 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22hp\" (UniqueName: \"kubernetes.io/projected/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-kube-api-access-k22hp\") pod \"payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t\" (UID: \"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" Nov 03 20:10:16.528410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.528368 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t\" (UID: \"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" Nov 03 20:10:16.540582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.540553 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22hp\" (UniqueName: \"kubernetes.io/projected/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-kube-api-access-k22hp\") pod \"payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t\" (UID: \"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" Nov 03 20:10:16.833700 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:16.833669 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" Nov 03 20:10:17.007408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:17.007139 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" event={"ID":"4de4b055-a669-4760-ba03-041291c506fd","Type":"ContainerStarted","Data":"b1c4a0757d3567e9ac47634d122b35a2841961058d97410a4e2a6ebd76c2cfcb"} Nov 03 20:10:17.007408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:17.007197 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" event={"ID":"4de4b055-a669-4760-ba03-041291c506fd","Type":"ContainerStarted","Data":"0976897cd8e00e948dac27c0c467af50a36a0501b99b9e44fa33a4f47d5c90c8"} Nov 03 20:10:17.007408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:17.007263 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" Nov 03 20:10:17.009151 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:17.009123 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" event={"ID":"250b873d-632c-4169-b357-77f6d8b9e9b6","Type":"ContainerStarted","Data":"d753e476a9da7265c502be8b7c91fcf092965660a7d3193cef0bdca44d3bf257"} Nov 03 20:10:17.009257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:17.009172 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" event={"ID":"250b873d-632c-4169-b357-77f6d8b9e9b6","Type":"ContainerStarted","Data":"627a83be6254c7676c30a4329aaf9075c91de4264af8017f393034259032651e"} Nov 03 20:10:17.009555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:17.009539 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" Nov 03 20:10:17.027244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:17.027201 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" podStartSLOduration=2.027189589 podStartE2EDuration="2.027189589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:17.025245076 +0000 UTC m=+795.400841822" watchObservedRunningTime="2026-11-03 20:10:17.027189589 +0000 UTC m=+795.402786335" Nov 03 20:10:17.045862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:17.045826 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" podStartSLOduration=2.04581602 podStartE2EDuration="2.04581602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:17.043169622 +0000 UTC m=+795.418766371" watchObservedRunningTime="2026-11-03 20:10:17.04581602 +0000 UTC m=+795.421412776" Nov 03 20:10:17.147022 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:17.146919 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t"] Nov 03 20:10:17.147859 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:10:17.147820 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1c7cba4_4279_4f48_9ecc_e48e6a0e2e58.slice/crio-1c19e4d6f2e951b493142cb129506c74adc80c46d523144cf95758c427ecde7c WatchSource:0}: Error finding container 1c19e4d6f2e951b493142cb129506c74adc80c46d523144cf95758c427ecde7c: Status 404 returned error can't find the container with id 1c19e4d6f2e951b493142cb129506c74adc80c46d523144cf95758c427ecde7c Nov 03 20:10:18.023081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:18.023045 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" event={"ID":"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58","Type":"ContainerStarted","Data":"93891d4861cb8f25d605df77b2b90208dbcd7462045025c23dc7224085eb9c23"} Nov 03 20:10:18.023673 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:18.023644 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" Nov 03 20:10:18.023821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:18.023685 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" event={"ID":"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58","Type":"ContainerStarted","Data":"1c19e4d6f2e951b493142cb129506c74adc80c46d523144cf95758c427ecde7c"} Nov 03 20:10:18.046664 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:18.046615 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" podStartSLOduration=2.046599348 podStartE2EDuration="2.046599348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:18.045372172 +0000 UTC m=+796.420968920" watchObservedRunningTime="2026-11-03 20:10:18.046599348 +0000 UTC m=+796.422196096" Nov 03 20:10:20.001755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:20.001708 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:10:21.966153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:21.965513 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" Nov 03 20:10:22.129979 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.126707 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg"] Nov 03 20:10:22.131877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.131851 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.143437 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.143407 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg"] Nov 03 20:10:22.144620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.143986 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a6f98b\"/\"default-dockercfg-7zd8f\"" Nov 03 20:10:22.144620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.144097 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a6f98b\"/\"kube-root-ca.crt\"" Nov 03 20:10:22.144620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.144125 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a6f98b\"/\"simulator-w5-a6f98b-kserve-self-signed-certs\"" Nov 03 20:10:22.144620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.144217 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a6f98b\"/\"openshift-service-ca.crt\"" Nov 03 20:10:22.197411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.196686 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b0efa8-3782-4cab-be92-a05fa383e808-tls-certs\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.197411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.196734 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-home\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.197411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.196767 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjg92\" (UniqueName: \"kubernetes.io/projected/04b0efa8-3782-4cab-be92-a05fa383e808-kube-api-access-pjg92\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.197411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.196831 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-tmp-dir\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.197411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.196860 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-model-cache\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.197411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.196914 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-dshm\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.298142 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.298106 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b0efa8-3782-4cab-be92-a05fa383e808-tls-certs\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.298142 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.298141 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-home\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.298415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.298169 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjg92\" (UniqueName: \"kubernetes.io/projected/04b0efa8-3782-4cab-be92-a05fa383e808-kube-api-access-pjg92\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.298415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.298251 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-tmp-dir\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.298415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.298285 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-model-cache\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.298415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.298330 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-dshm\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.298821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.298786 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-model-cache\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.298942 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.298858 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-home\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.299059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.299037 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-tmp-dir\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.301097 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.301072 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-dshm\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.301304 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.301273 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b0efa8-3782-4cab-be92-a05fa383e808-tls-certs\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.312465 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.312407 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjg92\" (UniqueName: \"kubernetes.io/projected/04b0efa8-3782-4cab-be92-a05fa383e808-kube-api-access-pjg92\") pod \"simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.468405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.468352 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:22.755799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.755761 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg"] Nov 03 20:10:22.757847 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:10:22.757819 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04b0efa8_3782_4cab_be92_a05fa383e808.slice/crio-e9b51d7cf82896d014612835462bd8d604e35e55d4faadf206bd647f76610b84 WatchSource:0}: Error finding container e9b51d7cf82896d014612835462bd8d604e35e55d4faadf206bd647f76610b84: Status 404 returned error can't find the container with id e9b51d7cf82896d014612835462bd8d604e35e55d4faadf206bd647f76610b84 Nov 03 20:10:22.973362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:22.973334 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" Nov 03 20:10:23.056030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:23.055983 4457 generic.go:356] "Generic (PLEG): container finished" podID="49cbfb85-a563-4265-828a-40ead144734b" containerID="fa891d3b4bad4521461ace4728816302f86a2b8ece0f8d5e701917cc39c88280" exitCode=0 Nov 03 20:10:23.056218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:23.056050 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" event={"ID":"49cbfb85-a563-4265-828a-40ead144734b","Type":"ContainerDied","Data":"fa891d3b4bad4521461ace4728816302f86a2b8ece0f8d5e701917cc39c88280"} Nov 03 20:10:23.056552 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:23.056518 4457 scope.go:122] "RemoveContainer" containerID="fa891d3b4bad4521461ace4728816302f86a2b8ece0f8d5e701917cc39c88280" Nov 03 20:10:23.057683 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:23.057651 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" event={"ID":"04b0efa8-3782-4cab-be92-a05fa383e808","Type":"ContainerStarted","Data":"e9b51d7cf82896d014612835462bd8d604e35e55d4faadf206bd647f76610b84"} Nov 03 20:10:23.058154 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:23.058120 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:23.058262 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:23.058211 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:23.058320 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:23.058276 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:24.106172 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:24.105454 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" event={"ID":"49cbfb85-a563-4265-828a-40ead144734b","Type":"ContainerStarted","Data":"f372bd35881b53c4a32db85199fc7910b04d16c8ae172f84953836040e55d9d8"} Nov 03 20:10:25.086194 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:25.086100 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:25.086571 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:25.086437 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:10:25.086571 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:25.086485 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" podUID="49cbfb85-a563-4265-828a-40ead144734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 20:10:25.134436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:25.127354 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" event={"ID":"04b0efa8-3782-4cab-be92-a05fa383e808","Type":"ContainerStarted","Data":"4a4f8e4fae0f4de61e6de911f894280be7689ae2b72fc94aa74dc2e3d090d592"} Nov 03 20:10:25.153761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:25.153699 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podStartSLOduration=0.905922968 podStartE2EDuration="3.153678446s" totalImagesPullingTime="2.247755478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:25.148772079 +0000 UTC m=+803.524368827" watchObservedRunningTime="2026-11-03 20:10:25.153678446 +0000 UTC m=+803.529275199" Nov 03 20:10:25.473412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:25.469527 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:26.088923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:26.088889 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:10:26.089254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:26.089209 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" podUID="49cbfb85-a563-4265-828a-40ead144734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 20:10:27.087411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:27.086777 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:10:27.087411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:27.086843 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" podUID="49cbfb85-a563-4265-828a-40ead144734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 20:10:28.032709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:28.032606 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" Nov 03 20:10:28.033825 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:28.033683 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" Nov 03 20:10:28.139412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:28.124495 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:10:28.139412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:28.124551 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" podUID="49cbfb85-a563-4265-828a-40ead144734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 20:10:29.031579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.031549 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" Nov 03 20:10:29.104410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.103964 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 03 20:10:29.104410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.104034 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" podUID="49cbfb85-a563-4265-828a-40ead144734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 03 20:10:29.424548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.424500 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq"] Nov 03 20:10:29.438412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.436987 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq"] Nov 03 20:10:29.438412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.437104 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.442705 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.442670 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d412cf\"/\"simulator-w6-d412cf-kserve-self-signed-certs\"" Nov 03 20:10:29.442861 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.442771 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d412cf\"/\"kube-root-ca.crt\"" Nov 03 20:10:29.442861 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.442814 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d412cf\"/\"openshift-service-ca.crt\"" Nov 03 20:10:29.442978 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.442866 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d412cf\"/\"default-dockercfg-l8wbg\"" Nov 03 20:10:29.580674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.580633 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-tmp-dir\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.581034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.581012 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a5de13-728b-4a75-8990-807d1d3811e4-tls-certs\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.581207 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.581189 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-home\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.581368 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.581353 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-dshm\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.581595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.581573 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5krm\" (UniqueName: \"kubernetes.io/projected/a5a5de13-728b-4a75-8990-807d1d3811e4-kube-api-access-h5krm\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.581816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.581793 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-model-cache\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.685066 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.684961 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5krm\" (UniqueName: \"kubernetes.io/projected/a5a5de13-728b-4a75-8990-807d1d3811e4-kube-api-access-h5krm\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.685762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.685736 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-model-cache\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.686457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.686439 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-tmp-dir\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.686929 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.686912 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a5de13-728b-4a75-8990-807d1d3811e4-tls-certs\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.687480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.687462 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-home\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.687944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.687926 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-dshm\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.698361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.687868 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-home\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.698572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.686836 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-tmp-dir\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.698712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.686324 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-model-cache\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.701205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.701178 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5krm\" (UniqueName: \"kubernetes.io/projected/a5a5de13-728b-4a75-8990-807d1d3811e4-kube-api-access-h5krm\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.708041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.708015 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a5de13-728b-4a75-8990-807d1d3811e4-tls-certs\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.710684 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.710636 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-dshm\") pod \"simulator-w6-d412cf-kserve-b979cf7bc-zfwnq\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:29.765412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:29.763649 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:30.096322 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:30.096291 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:30.096605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:30.096582 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:30.097240 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:30.097221 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:10:30.118180 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:30.118149 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq"] Nov 03 20:10:30.119187 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:10:30.119161 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5a5de13_728b_4a75_8990_807d1d3811e4.slice/crio-ea42f29c37048574987e18c348b015b9f4fea853009dd05b59245a69df9507ab WatchSource:0}: Error finding container ea42f29c37048574987e18c348b015b9f4fea853009dd05b59245a69df9507ab: Status 404 returned error can't find the container with id ea42f29c37048574987e18c348b015b9f4fea853009dd05b59245a69df9507ab Nov 03 20:10:30.215030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:30.214997 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" event={"ID":"a5a5de13-728b-4a75-8990-807d1d3811e4","Type":"ContainerStarted","Data":"ea42f29c37048574987e18c348b015b9f4fea853009dd05b59245a69df9507ab"} Nov 03 20:10:31.223016 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:31.222922 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" event={"ID":"a5a5de13-728b-4a75-8990-807d1d3811e4","Type":"ContainerStarted","Data":"a4863bcc5fe6ab9849a22ebb0926c64f321e44e4d6b2a3949c215f2ef1b85d25"} Nov 03 20:10:31.247129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:31.247080 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podStartSLOduration=2.247063711 podStartE2EDuration="2.247063711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:31.246248575 +0000 UTC m=+809.621845323" watchObservedRunningTime="2026-11-03 20:10:31.247063711 +0000 UTC m=+809.622660664" Nov 03 20:10:31.764754 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:31.764712 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:32.469093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:32.469055 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:32.504306 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:32.504278 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:33.247361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.247326 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:10:33.694712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.694674 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9"] Nov 03 20:10:33.708153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.708120 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.709630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.709602 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9"] Nov 03 20:10:33.718171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.718140 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-90ef9c\"/\"openshift-service-ca.crt\"" Nov 03 20:10:33.718502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.718480 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-90ef9c\"/\"kube-root-ca.crt\"" Nov 03 20:10:33.721100 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.721076 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-90ef9c\"/\"simulator-w2-90ef9c-kserve-self-signed-certs\"" Nov 03 20:10:33.721325 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.721306 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-90ef9c\"/\"default-dockercfg-db8p9\"" Nov 03 20:10:33.744600 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.744563 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-model-cache\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.744776 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.744618 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-tmp-dir\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.744776 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.744739 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24f9087-0bd1-426c-8182-86be5a40418a-tls-certs\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.744880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.744780 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-home\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.744880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.744853 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-dshm\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.744979 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.744941 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7s55\" (UniqueName: \"kubernetes.io/projected/d24f9087-0bd1-426c-8182-86be5a40418a-kube-api-access-j7s55\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.846334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.846296 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24f9087-0bd1-426c-8182-86be5a40418a-tls-certs\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.846549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.846359 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-home\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.846549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.846415 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-dshm\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.846549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.846495 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7s55\" (UniqueName: \"kubernetes.io/projected/d24f9087-0bd1-426c-8182-86be5a40418a-kube-api-access-j7s55\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.846735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.846647 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-model-cache\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.846735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.846684 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-tmp-dir\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.846862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.846836 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-home\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.847200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.847171 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-model-cache\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.847335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.847240 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-tmp-dir\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.849448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.849422 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-dshm\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.849598 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.849578 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24f9087-0bd1-426c-8182-86be5a40418a-tls-certs\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.858591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.858568 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7s55\" (UniqueName: \"kubernetes.io/projected/d24f9087-0bd1-426c-8182-86be5a40418a-kube-api-access-j7s55\") pod \"simulator-w2-90ef9c-kserve-644c9585b7-t8pz9\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:33.875076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.874718 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:10:33.875076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:33.874794 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:10:34.026359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:34.026261 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:34.313115 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:34.313080 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9"] Nov 03 20:10:34.317192 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:10:34.317148 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd24f9087_0bd1_426c_8182_86be5a40418a.slice/crio-e65500eb59911cc8e1c051fe240d97080e7e38aae34b397c383e0c3c431c8c12 WatchSource:0}: Error finding container e65500eb59911cc8e1c051fe240d97080e7e38aae34b397c383e0c3c431c8c12: Status 404 returned error can't find the container with id e65500eb59911cc8e1c051fe240d97080e7e38aae34b397c383e0c3c431c8c12 Nov 03 20:10:35.248943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:35.248861 4457 generic.go:356] "Generic (PLEG): container finished" podID="942423a8-2c01-4ef8-82df-cf8892de3585" containerID="bd1119a197ada5445a2ce8cdb33836c21d6ee8ab27178de9b21ead66762627cb" exitCode=0 Nov 03 20:10:35.249379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:35.248937 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" event={"ID":"942423a8-2c01-4ef8-82df-cf8892de3585","Type":"ContainerDied","Data":"bd1119a197ada5445a2ce8cdb33836c21d6ee8ab27178de9b21ead66762627cb"} Nov 03 20:10:35.249379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:35.249369 4457 scope.go:122] "RemoveContainer" containerID="bd1119a197ada5445a2ce8cdb33836c21d6ee8ab27178de9b21ead66762627cb" Nov 03 20:10:35.250809 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:35.250790 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" event={"ID":"d24f9087-0bd1-426c-8182-86be5a40418a","Type":"ContainerStarted","Data":"9df1a760a04b92a6546eadf08c55349bb12dcefbe58719080e97f6a3caebab9a"} Nov 03 20:10:35.250903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:35.250819 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" event={"ID":"d24f9087-0bd1-426c-8182-86be5a40418a","Type":"ContainerStarted","Data":"e65500eb59911cc8e1c051fe240d97080e7e38aae34b397c383e0c3c431c8c12"} Nov 03 20:10:35.250952 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:35.250889 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:35.250992 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:35.250975 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:35.251075 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:35.251059 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:10:35.300838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:35.300776 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podStartSLOduration=2.300756451 podStartE2EDuration="2.300756451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:35.298985432 +0000 UTC m=+813.674582191" watchObservedRunningTime="2026-11-03 20:10:35.300756451 +0000 UTC m=+813.676353198" Nov 03 20:10:36.026976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.026939 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:36.147357 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:10:36.147315 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:10:36.272283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.272248 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm"] Nov 03 20:10:36.313560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.313524 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" event={"ID":"942423a8-2c01-4ef8-82df-cf8892de3585","Type":"ContainerStarted","Data":"4c334a09f7b1cde8c49a9c999772d1cb646484e8ea8cbea90388ffa69e2f01a7"} Nov 03 20:10:36.313826 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.313805 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm"] Nov 03 20:10:36.314270 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.314242 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.322862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.322842 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1b190\"/\"default-dockercfg-5lshz\"" Nov 03 20:10:36.324604 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.323854 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b1b190\"/\"kube-root-ca.crt\"" Nov 03 20:10:36.324604 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.323957 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-b1b190\"/\"openshift-service-ca.crt\"" Nov 03 20:10:36.326969 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.326641 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1b190\"/\"simulator-w0-b1b190-kserve-self-signed-certs\"" Nov 03 20:10:36.399665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.399628 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-dshm\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.399920 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.399903 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-home\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.400066 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.400051 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qfts\" (UniqueName: \"kubernetes.io/projected/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-kube-api-access-7qfts\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.400270 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.400256 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-model-cache\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.400516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.400498 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tmp-dir\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.400764 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.400749 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tls-certs\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.505411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.502437 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-model-cache\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.505411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.502511 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tmp-dir\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.505411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.502557 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tls-certs\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.505411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.502647 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-dshm\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.505411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.502688 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-home\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.505411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.502732 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qfts\" (UniqueName: \"kubernetes.io/projected/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-kube-api-access-7qfts\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.505411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.503557 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-model-cache\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.505411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.503808 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tmp-dir\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.506382 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.506347 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-home\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.526062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.524076 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qfts\" (UniqueName: \"kubernetes.io/projected/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-kube-api-access-7qfts\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.526062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.524616 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tls-certs\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.529706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.529641 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-dshm\") pod \"simulator-w0-b1b190-kserve-7f9d69dd-4vcqm\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.629744 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.629645 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:36.942606 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:36.942577 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm"] Nov 03 20:10:36.944002 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:10:36.943974 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea3405cc_6fa5_459c_a7b2_189a1b09c6d5.slice/crio-de4fd98da4227dcb3c5d153f2b8778a18a8dec516253cce5157c4fab1a7237e2 WatchSource:0}: Error finding container de4fd98da4227dcb3c5d153f2b8778a18a8dec516253cce5157c4fab1a7237e2: Status 404 returned error can't find the container with id de4fd98da4227dcb3c5d153f2b8778a18a8dec516253cce5157c4fab1a7237e2 Nov 03 20:10:37.001238 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:37.001206 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:37.001460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:37.001400 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:10:37.001536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:37.001472 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 20:10:37.308355 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:37.308314 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" event={"ID":"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5","Type":"ContainerStarted","Data":"92871e60191a1ad43dbde6054dc33eebd1ac1fd2b4f5d720c04786783baa9c5e"} Nov 03 20:10:37.308890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:37.308371 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" event={"ID":"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5","Type":"ContainerStarted","Data":"de4fd98da4227dcb3c5d153f2b8778a18a8dec516253cce5157c4fab1a7237e2"} Nov 03 20:10:37.369412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:37.369167 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podStartSLOduration=1.369133411 podStartE2EDuration="1.369133411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:37.362925618 +0000 UTC m=+815.738522368" watchObservedRunningTime="2026-11-03 20:10:37.369133411 +0000 UTC m=+815.744730160" Nov 03 20:10:37.630462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:37.630055 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:38.001606 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.001510 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:10:38.001606 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.001582 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 20:10:38.754886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.754849 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r"] Nov 03 20:10:38.773409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.758221 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.777409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.774167 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8013d9\"/\"simulator-w1-8013d9-kserve-self-signed-certs\"" Nov 03 20:10:38.777409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.774585 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8013d9\"/\"openshift-service-ca.crt\"" Nov 03 20:10:38.777409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.775206 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8013d9\"/\"kube-root-ca.crt\"" Nov 03 20:10:38.777409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.775315 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8013d9\"/\"default-dockercfg-cvfv5\"" Nov 03 20:10:38.800564 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.800534 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r"] Nov 03 20:10:38.825682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.825190 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq94q\" (UniqueName: \"kubernetes.io/projected/555064e0-9ca2-47de-b1c0-8473f3729219-kube-api-access-dq94q\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.825682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.825254 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/555064e0-9ca2-47de-b1c0-8473f3729219-tls-certs\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.825682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.825287 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-dshm\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.825682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.825314 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-model-cache\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.825682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.825372 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-home\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.825682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.825510 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-tmp-dir\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.926600 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.926566 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-tmp-dir\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.926821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.926625 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq94q\" (UniqueName: \"kubernetes.io/projected/555064e0-9ca2-47de-b1c0-8473f3729219-kube-api-access-dq94q\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.926821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.926665 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/555064e0-9ca2-47de-b1c0-8473f3729219-tls-certs\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.926821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.926696 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-dshm\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.926821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.926719 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-model-cache\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.926821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.926771 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-home\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.927093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.927013 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-tmp-dir\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.927171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.927152 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-home\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.927859 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.927836 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-model-cache\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.930007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.929986 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/555064e0-9ca2-47de-b1c0-8473f3729219-tls-certs\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.932632 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.932610 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-dshm\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:38.960406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:38.955165 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq94q\" (UniqueName: \"kubernetes.io/projected/555064e0-9ca2-47de-b1c0-8473f3729219-kube-api-access-dq94q\") pod \"simulator-w1-8013d9-kserve-7d969b758f-kmh4r\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:39.002225 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:39.001830 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:10:39.002225 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:39.001893 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 20:10:39.118825 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:39.118786 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:39.438932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:39.438839 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r"] Nov 03 20:10:39.776468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:39.768436 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:39.800414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:39.800134 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:40.001423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:40.001283 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:10:40.001423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:40.001346 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 20:10:40.330006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:40.329859 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" event={"ID":"555064e0-9ca2-47de-b1c0-8473f3729219","Type":"ContainerStarted","Data":"69a70f88ecb774f0acc80679a1fc73a3e9b5e9236ca635484edece286823ecff"} Nov 03 20:10:40.330006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:40.329917 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" event={"ID":"555064e0-9ca2-47de-b1c0-8473f3729219","Type":"ContainerStarted","Data":"0d74dc6b0d869b3fb3d1b9a0bab939395b24a8f08101eb4c5736797a702c7a04"} Nov 03 20:10:40.342475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:40.342448 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:10:40.352195 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:40.352155 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podStartSLOduration=2.352143665 podStartE2EDuration="2.352143665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:40.351693632 +0000 UTC m=+818.727290379" watchObservedRunningTime="2026-11-03 20:10:40.352143665 +0000 UTC m=+818.727740412" Nov 03 20:10:41.001967 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:41.001930 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Nov 03 20:10:41.002372 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:41.001987 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Nov 03 20:10:41.119249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:41.119205 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:42.005160 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:42.005128 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:42.005681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:42.005506 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:42.006377 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:42.006357 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:10:44.027414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:44.027291 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:44.046222 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:44.046185 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:44.373218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:44.373186 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:10:46.630796 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:46.630761 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:46.645302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:46.645275 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:47.389855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:47.389823 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:10:49.119434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:49.119381 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:49.134676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:49.134651 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:49.402831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:49.402744 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:10:51.766821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.766788 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h"] Nov 03 20:10:51.772097 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.771257 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.779130 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.779101 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-62221d\"/\"openshift-service-ca.crt\"" Nov 03 20:10:51.779276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.779162 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-62221d\"/\"simulator-w3-62221d-kserve-self-signed-certs\"" Nov 03 20:10:51.779276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.779186 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-62221d\"/\"kube-root-ca.crt\"" Nov 03 20:10:51.779382 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.779301 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-62221d\"/\"default-dockercfg-d2pzc\"" Nov 03 20:10:51.782308 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.782283 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h"] Nov 03 20:10:51.872068 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.872025 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4254p\" (UniqueName: \"kubernetes.io/projected/5d37f893-d432-4047-8b6b-141094b1524f-kube-api-access-4254p\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.872068 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.872065 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-model-cache\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.872359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.872148 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-tmp-dir\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.872359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.872195 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-dshm\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.872359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.872296 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-home\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.872359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.872351 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d37f893-d432-4047-8b6b-141094b1524f-tls-certs\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.973180 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.973142 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-tmp-dir\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.973381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.973201 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-dshm\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.973381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.973264 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-home\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.973381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.973329 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d37f893-d432-4047-8b6b-141094b1524f-tls-certs\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.973555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.973406 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4254p\" (UniqueName: \"kubernetes.io/projected/5d37f893-d432-4047-8b6b-141094b1524f-kube-api-access-4254p\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.973555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.973432 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-model-cache\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.973727 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.973702 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-tmp-dir\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.974162 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.974137 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-home\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.974529 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.974502 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-model-cache\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.976626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.976601 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-dshm\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.976874 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.976850 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d37f893-d432-4047-8b6b-141094b1524f-tls-certs\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:51.988294 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:51.988265 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4254p\" (UniqueName: \"kubernetes.io/projected/5d37f893-d432-4047-8b6b-141094b1524f-kube-api-access-4254p\") pod \"simulator-w3-62221d-kserve-547bf46c9c-qtv4h\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:52.089820 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:52.089789 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:10:52.390082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:52.389787 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h"] Nov 03 20:10:52.390824 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:10:52.390797 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d37f893_d432_4047_8b6b_141094b1524f.slice/crio-c6c312021c1a8a90e5f7f0d23402af51db335b937d8452b5917f085ababf25c0 WatchSource:0}: Error finding container c6c312021c1a8a90e5f7f0d23402af51db335b937d8452b5917f085ababf25c0: Status 404 returned error can't find the container with id c6c312021c1a8a90e5f7f0d23402af51db335b937d8452b5917f085ababf25c0 Nov 03 20:10:52.418326 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:52.418281 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" event={"ID":"5d37f893-d432-4047-8b6b-141094b1524f","Type":"ContainerStarted","Data":"c6c312021c1a8a90e5f7f0d23402af51db335b937d8452b5917f085ababf25c0"} Nov 03 20:10:53.425374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:53.425338 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" event={"ID":"5d37f893-d432-4047-8b6b-141094b1524f","Type":"ContainerStarted","Data":"1202e7c5ffe7a6e731deaed54e537188b37a2d68888eb5e3a3f4a70feba78f94"} Nov 03 20:10:53.447036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:53.446963 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podStartSLOduration=2.446950259 podStartE2EDuration="2.446950259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:10:53.444856705 +0000 UTC m=+831.820453456" watchObservedRunningTime="2026-11-03 20:10:53.446950259 +0000 UTC m=+831.822547006" Nov 03 20:10:54.090545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:10:54.090506 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:11:02.090418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.090365 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:11:02.108261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.108233 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:11:02.169084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.169052 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8"] Nov 03 20:11:02.175116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.175089 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.183152 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.182965 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-90ef9c\"/\"premium-w2-90ef9c-kserve-self-signed-certs\"" Nov 03 20:11:02.189487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.188164 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8"] Nov 03 20:11:02.298415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.294372 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-home\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.298415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.294465 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrql9\" (UniqueName: \"kubernetes.io/projected/4a9b8725-2131-4b8d-9eae-e2ef60be3627-kube-api-access-lrql9\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.298415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.294626 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tmp-dir\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.298415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.294698 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tls-certs\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.298415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.294747 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-dshm\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.298415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.294781 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-model-cache\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.402459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.398827 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tmp-dir\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.402459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.398909 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tls-certs\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.402459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.398941 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-dshm\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.402459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.398977 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-model-cache\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.402459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.399097 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-home\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.402459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.399136 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrql9\" (UniqueName: \"kubernetes.io/projected/4a9b8725-2131-4b8d-9eae-e2ef60be3627-kube-api-access-lrql9\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.402459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.399899 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tmp-dir\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.406186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.404974 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-model-cache\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.406186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.405451 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-home\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.408024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.407988 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tls-certs\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.408144 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.408030 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-dshm\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.412185 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.412160 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrql9\" (UniqueName: \"kubernetes.io/projected/4a9b8725-2131-4b8d-9eae-e2ef60be3627-kube-api-access-lrql9\") pod \"premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.494719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.494684 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:02.499799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.499777 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:11:02.805282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:02.805240 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8"] Nov 03 20:11:02.813752 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:11:02.813717 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a9b8725_2131_4b8d_9eae_e2ef60be3627.slice/crio-5f72cdbce641982a5caf4b3cf7b0973a542b1e5cd367c0d78fd5b848987cb7c6 WatchSource:0}: Error finding container 5f72cdbce641982a5caf4b3cf7b0973a542b1e5cd367c0d78fd5b848987cb7c6: Status 404 returned error can't find the container with id 5f72cdbce641982a5caf4b3cf7b0973a542b1e5cd367c0d78fd5b848987cb7c6 Nov 03 20:11:03.494944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:03.494905 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" event={"ID":"4a9b8725-2131-4b8d-9eae-e2ef60be3627","Type":"ContainerStarted","Data":"39246feb65cb2de0354f0d0eb827bcb44911089f5050873568127428c3e6cf36"} Nov 03 20:11:03.495443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:03.494957 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" event={"ID":"4a9b8725-2131-4b8d-9eae-e2ef60be3627","Type":"ContainerStarted","Data":"5f72cdbce641982a5caf4b3cf7b0973a542b1e5cd367c0d78fd5b848987cb7c6"} Nov 03 20:11:03.517771 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:03.517722 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podStartSLOduration=1.5177084189999999 podStartE2EDuration="1.517708419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:03.514799194 +0000 UTC m=+841.890395942" watchObservedRunningTime="2026-11-03 20:11:03.517708419 +0000 UTC m=+841.893305166" Nov 03 20:11:04.495552 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:04.495517 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:06.147788 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:11:06.147744 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:11:12.495453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:12.495416 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:12.518880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:12.518842 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:12.563993 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:12.563961 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:11:13.880985 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:13.880952 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q"] Nov 03 20:11:13.886318 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:13.886295 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:13.897217 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:13.897183 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3aba31\"/\"default-dockercfg-rcqfz\"" Nov 03 20:11:13.897971 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:13.897947 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-3aba31\"/\"premium-w4-3aba31-kserve-self-signed-certs\"" Nov 03 20:11:13.900345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:13.898208 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3aba31\"/\"kube-root-ca.crt\"" Nov 03 20:11:13.900345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:13.898328 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-3aba31\"/\"openshift-service-ca.crt\"" Nov 03 20:11:13.903900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:13.903873 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q"] Nov 03 20:11:13.945846 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:13.945813 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-dshm\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:13.946171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:13.946145 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f4267e-0c59-4326-972b-b0925e478929-tls-certs\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:13.946281 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:13.946258 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-home\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:13.946343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:13.946323 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-tmp-dir\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:13.946419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:13.946358 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-model-cache\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:13.946487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:13.946427 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ss95\" (UniqueName: \"kubernetes.io/projected/f3f4267e-0c59-4326-972b-b0925e478929-kube-api-access-9ss95\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.047283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.047242 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-dshm\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.048260 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.047612 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f4267e-0c59-4326-972b-b0925e478929-tls-certs\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.048480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.048462 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-home\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.048617 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.048604 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-tmp-dir\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.048726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.048713 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-model-cache\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.048862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.048848 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ss95\" (UniqueName: \"kubernetes.io/projected/f3f4267e-0c59-4326-972b-b0925e478929-kube-api-access-9ss95\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.050944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.050077 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-tmp-dir\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.050944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.050903 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-model-cache\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.052246 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.051376 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-home\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.053934 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.053911 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-dshm\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.067461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.061601 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ss95\" (UniqueName: \"kubernetes.io/projected/f3f4267e-0c59-4326-972b-b0925e478929-kube-api-access-9ss95\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.069268 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.069219 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f4267e-0c59-4326-972b-b0925e478929-tls-certs\") pod \"premium-w4-3aba31-kserve-bc7889789-vsl6q\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.239334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.239242 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:14.531398 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:11:14.531355 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3f4267e_0c59_4326_972b_b0925e478929.slice/crio-ee14ea6bf6f6b6302e14466f6f60949d71fe4e10e143a8a7facd973e748ab962 WatchSource:0}: Error finding container ee14ea6bf6f6b6302e14466f6f60949d71fe4e10e143a8a7facd973e748ab962: Status 404 returned error can't find the container with id ee14ea6bf6f6b6302e14466f6f60949d71fe4e10e143a8a7facd973e748ab962 Nov 03 20:11:14.532120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.532088 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q"] Nov 03 20:11:14.562833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:14.562767 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" event={"ID":"f3f4267e-0c59-4326-972b-b0925e478929","Type":"ContainerStarted","Data":"ee14ea6bf6f6b6302e14466f6f60949d71fe4e10e143a8a7facd973e748ab962"} Nov 03 20:11:15.571018 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:15.570986 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" event={"ID":"f3f4267e-0c59-4326-972b-b0925e478929","Type":"ContainerStarted","Data":"74bcd017eff3d35badc298cc237c4ab009a62f20203a14cf6667da65487b1070"} Nov 03 20:11:15.594848 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:15.594799 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podStartSLOduration=2.594784651 podStartE2EDuration="2.594784651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:15.5923139 +0000 UTC m=+853.967910655" watchObservedRunningTime="2026-11-03 20:11:15.594784651 +0000 UTC m=+853.970381399" Nov 03 20:11:16.239862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:16.239820 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:24.240103 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:24.240071 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:24.252783 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:24.252757 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:24.644454 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:24.644426 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:11:36.145486 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:11:36.145439 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:11:45.660214 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.660177 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh"] Nov 03 20:11:45.665676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.665649 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.669512 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.669486 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-d412cf-openshift-default-dockercfg-qrnd5\"" Nov 03 20:11:45.676216 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.676194 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh"] Nov 03 20:11:45.790333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.790289 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.790545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.790348 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.790545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.790452 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf12593-ef2a-4848-be72-ce57d759c2e9-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.790545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.790503 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.790665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.790618 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.790665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.790656 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.790737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.790686 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5v78\" (UniqueName: \"kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-kube-api-access-p5v78\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.790823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.790806 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.790867 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.790834 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.892339 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.892301 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.892557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.892356 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.892557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.892400 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf12593-ef2a-4848-be72-ce57d759c2e9-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.892557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.892436 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.892557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.892503 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.892557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.892541 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.892838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.892580 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5v78\" (UniqueName: \"kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-kube-api-access-p5v78\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.892838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.892658 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.892947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.892915 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.892999 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.892938 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.893271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.892854 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.893379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.893137 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.893468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.893192 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.893648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.893625 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf12593-ef2a-4848-be72-ce57d759c2e9-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.894813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.894790 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.895257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.895240 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.907925 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.907903 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.908102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.908085 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5v78\" (UniqueName: \"kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-kube-api-access-p5v78\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:45.981761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:45.981677 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:46.271374 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:11:46.271345 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf12593_ef2a_4848_be72_ce57d759c2e9.slice/crio-4add2406c38992eaff3bd74ce8fcc867b6fc71515992fdd3bf8e8c17b40dcf51 WatchSource:0}: Error finding container 4add2406c38992eaff3bd74ce8fcc867b6fc71515992fdd3bf8e8c17b40dcf51: Status 404 returned error can't find the container with id 4add2406c38992eaff3bd74ce8fcc867b6fc71515992fdd3bf8e8c17b40dcf51 Nov 03 20:11:46.271549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:46.271523 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh"] Nov 03 20:11:46.274620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:46.274580 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:46.274718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:46.274687 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:46.274758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:46.274733 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:11:46.790994 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:46.790956 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" event={"ID":"6cf12593-ef2a-4848-be72-ce57d759c2e9","Type":"ContainerStarted","Data":"b8049778edcb9e1b90bdca5cb8dcf3f227004ba34cc059918b1ed4a6b7e23b32"} Nov 03 20:11:46.791433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:46.791016 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" event={"ID":"6cf12593-ef2a-4848-be72-ce57d759c2e9","Type":"ContainerStarted","Data":"4add2406c38992eaff3bd74ce8fcc867b6fc71515992fdd3bf8e8c17b40dcf51"} Nov 03 20:11:46.814662 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:46.814605 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" podStartSLOduration=1.8145881670000001 podStartE2EDuration="1.814588167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:11:46.812761752 +0000 UTC m=+885.188358499" watchObservedRunningTime="2026-11-03 20:11:46.814588167 +0000 UTC m=+885.190184915" Nov 03 20:11:46.982761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:46.982722 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:47.988484 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:47.988180 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:11:47.988484 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:47.988248 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" podUID="6cf12593-ef2a-4848-be72-ce57d759c2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 20:11:48.982147 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:48.982105 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:11:48.982520 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:48.982487 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" podUID="6cf12593-ef2a-4848-be72-ce57d759c2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 20:11:49.983823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:49.983781 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:11:49.984456 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:49.984423 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" podUID="6cf12593-ef2a-4848-be72-ce57d759c2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 20:11:50.986823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:50.986784 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:11:50.987576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:50.987538 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" podUID="6cf12593-ef2a-4848-be72-ce57d759c2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 20:11:51.983778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:51.983740 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 03 20:11:51.984098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:51.984054 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" podUID="6cf12593-ef2a-4848-be72-ce57d759c2e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 03 20:11:52.986057 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:52.986023 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:52.986567 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:52.986420 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:11:52.987103 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:11:52.987084 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:12:01.826381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:01.826344 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:12:01.836946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:01.836923 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:12:01.847496 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:01.847474 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:12:01.851430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:01.851377 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:12:04.799706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:04.799662 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5"] Nov 03 20:12:04.803915 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:04.803835 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" Nov 03 20:12:04.807683 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:04.807657 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:12:04.809107 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:04.809088 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:12:04.813674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:04.813651 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5"] Nov 03 20:12:04.905590 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:04.905554 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b4ws\" (UniqueName: \"kubernetes.io/projected/2b11d16e-96b3-43c4-8a74-2d3536a42f67-kube-api-access-8b4ws\") pod \"maas-api-revoke-sub-727f0656-kk7c5\" (UID: \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" Nov 03 20:12:04.905805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:04.905678 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-727f0656-kk7c5\" (UID: \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" Nov 03 20:12:04.905805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:04.905745 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-727f0656-kk7c5\" (UID: \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" Nov 03 20:12:05.007355 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:05.007312 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4ws\" (UniqueName: \"kubernetes.io/projected/2b11d16e-96b3-43c4-8a74-2d3536a42f67-kube-api-access-8b4ws\") pod \"maas-api-revoke-sub-727f0656-kk7c5\" (UID: \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" Nov 03 20:12:05.007531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:05.007374 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-727f0656-kk7c5\" (UID: \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" Nov 03 20:12:05.007531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:05.007433 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-727f0656-kk7c5\" (UID: \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" Nov 03 20:12:05.008037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:05.008016 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-727f0656-kk7c5\" (UID: \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" Nov 03 20:12:05.009772 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:05.009747 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-727f0656-kk7c5\" (UID: \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" Nov 03 20:12:05.019884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:05.019858 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4ws\" (UniqueName: \"kubernetes.io/projected/2b11d16e-96b3-43c4-8a74-2d3536a42f67-kube-api-access-8b4ws\") pod \"maas-api-revoke-sub-727f0656-kk7c5\" (UID: \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" Nov 03 20:12:05.124200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:05.124115 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" Nov 03 20:12:05.410980 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:05.410940 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5"] Nov 03 20:12:05.412078 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:12:05.411987 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b11d16e_96b3_43c4_8a74_2d3536a42f67.slice/crio-0063b9c1cc815e68bf38a2e7e5509b177111b6e7e5d309f56f9192941190e50c WatchSource:0}: Error finding container 0063b9c1cc815e68bf38a2e7e5509b177111b6e7e5d309f56f9192941190e50c: Status 404 returned error can't find the container with id 0063b9c1cc815e68bf38a2e7e5509b177111b6e7e5d309f56f9192941190e50c Nov 03 20:12:05.924023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:05.923981 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" event={"ID":"2b11d16e-96b3-43c4-8a74-2d3536a42f67","Type":"ContainerStarted","Data":"0063b9c1cc815e68bf38a2e7e5509b177111b6e7e5d309f56f9192941190e50c"} Nov 03 20:12:06.145626 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:12:06.145592 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:12:08.349227 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.349193 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c"] Nov 03 20:12:08.353854 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.353376 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" Nov 03 20:12:08.368370 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.367533 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c"] Nov 03 20:12:08.444752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.444719 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx4lf\" (UniqueName: \"kubernetes.io/projected/669b4131-2371-4b01-b932-7035e230d493-kube-api-access-jx4lf\") pod \"maas-api-revoke-sub-247ddf7f-r9j7c\" (UID: \"669b4131-2371-4b01-b932-7035e230d493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" Nov 03 20:12:08.444942 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.444789 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669b4131-2371-4b01-b932-7035e230d493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-247ddf7f-r9j7c\" (UID: \"669b4131-2371-4b01-b932-7035e230d493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" Nov 03 20:12:08.445012 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.444969 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669b4131-2371-4b01-b932-7035e230d493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-247ddf7f-r9j7c\" (UID: \"669b4131-2371-4b01-b932-7035e230d493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" Nov 03 20:12:08.547025 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.545974 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx4lf\" (UniqueName: \"kubernetes.io/projected/669b4131-2371-4b01-b932-7035e230d493-kube-api-access-jx4lf\") pod \"maas-api-revoke-sub-247ddf7f-r9j7c\" (UID: \"669b4131-2371-4b01-b932-7035e230d493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" Nov 03 20:12:08.547025 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.546079 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669b4131-2371-4b01-b932-7035e230d493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-247ddf7f-r9j7c\" (UID: \"669b4131-2371-4b01-b932-7035e230d493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" Nov 03 20:12:08.547025 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.546157 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669b4131-2371-4b01-b932-7035e230d493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-247ddf7f-r9j7c\" (UID: \"669b4131-2371-4b01-b932-7035e230d493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" Nov 03 20:12:08.547025 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.546959 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669b4131-2371-4b01-b932-7035e230d493-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-247ddf7f-r9j7c\" (UID: \"669b4131-2371-4b01-b932-7035e230d493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" Nov 03 20:12:08.561451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.549430 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669b4131-2371-4b01-b932-7035e230d493-maas-api-xks-ca\") pod \"maas-api-revoke-sub-247ddf7f-r9j7c\" (UID: \"669b4131-2371-4b01-b932-7035e230d493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" Nov 03 20:12:08.569420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.563719 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx4lf\" (UniqueName: \"kubernetes.io/projected/669b4131-2371-4b01-b932-7035e230d493-kube-api-access-jx4lf\") pod \"maas-api-revoke-sub-247ddf7f-r9j7c\" (UID: \"669b4131-2371-4b01-b932-7035e230d493\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" Nov 03 20:12:08.673327 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.673248 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" Nov 03 20:12:08.973761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:08.973682 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c"] Nov 03 20:12:08.977742 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:12:08.977708 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669b4131_2371_4b01_b932_7035e230d493.slice/crio-390f6b69836a76c0bcb00c1f27fc80e625eed6fb0ea75b1850ef4ef4da84d1e3 WatchSource:0}: Error finding container 390f6b69836a76c0bcb00c1f27fc80e625eed6fb0ea75b1850ef4ef4da84d1e3: Status 404 returned error can't find the container with id 390f6b69836a76c0bcb00c1f27fc80e625eed6fb0ea75b1850ef4ef4da84d1e3 Nov 03 20:12:09.952846 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:09.952756 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" event={"ID":"2b11d16e-96b3-43c4-8a74-2d3536a42f67","Type":"ContainerStarted","Data":"82a1aaeca0666d4efb176777434b53953f4834a143ad186b2d8c7f6e8bc62d65"} Nov 03 20:12:09.954074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:09.954049 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" event={"ID":"669b4131-2371-4b01-b932-7035e230d493","Type":"ContainerStarted","Data":"390f6b69836a76c0bcb00c1f27fc80e625eed6fb0ea75b1850ef4ef4da84d1e3"} Nov 03 20:12:09.974145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:09.974101 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" podStartSLOduration=2.043369182 podStartE2EDuration="5.974088444s" totalImagesPullingTime="3.930719262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:09.972533535 +0000 UTC m=+908.348130284" watchObservedRunningTime="2026-11-03 20:12:09.974088444 +0000 UTC m=+908.349685190" Nov 03 20:12:10.964424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:10.964317 4457 generic.go:356] "Generic (PLEG): container finished" podID="2b11d16e-96b3-43c4-8a74-2d3536a42f67" containerID="82a1aaeca0666d4efb176777434b53953f4834a143ad186b2d8c7f6e8bc62d65" exitCode=0 Nov 03 20:12:10.964782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:10.964439 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" event={"ID":"2b11d16e-96b3-43c4-8a74-2d3536a42f67","Type":"ContainerDied","Data":"82a1aaeca0666d4efb176777434b53953f4834a143ad186b2d8c7f6e8bc62d65"} Nov 03 20:12:11.974025 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:11.973987 4457 generic.go:356] "Generic (PLEG): container finished" podID="669b4131-2371-4b01-b932-7035e230d493" containerID="2814324eedcc19edc699eebf2c13a8b5b76b28b84aadc69e3af5cb52c92ef3e8" exitCode=0 Nov 03 20:12:11.974532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:11.974081 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" event={"ID":"669b4131-2371-4b01-b932-7035e230d493","Type":"ContainerDied","Data":"2814324eedcc19edc699eebf2c13a8b5b76b28b84aadc69e3af5cb52c92ef3e8"} Nov 03 20:12:12.132059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:12.132034 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" Nov 03 20:12:12.296201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:12.296169 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b11d16e-96b3-43c4-8a74-2d3536a42f67-kube-api-access-8b4ws\" (UniqueName: \"kubernetes.io/projected/2b11d16e-96b3-43c4-8a74-2d3536a42f67-kube-api-access-8b4ws\") pod \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\" (UID: \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\") " Nov 03 20:12:12.296369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:12.296295 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-xks-ca\") pod \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\" (UID: \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\") " Nov 03 20:12:12.296889 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:12.296433 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-openshift-ca\") pod \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\" (UID: \"2b11d16e-96b3-43c4-8a74-2d3536a42f67\") " Nov 03 20:12:12.297218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:12.297188 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-openshift-ca" pod "2b11d16e-96b3-43c4-8a74-2d3536a42f67" (UID: "2b11d16e-96b3-43c4-8a74-2d3536a42f67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:12.299145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:12.299117 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b11d16e-96b3-43c4-8a74-2d3536a42f67-kube-api-access-8b4ws" pod "2b11d16e-96b3-43c4-8a74-2d3536a42f67" (UID: "2b11d16e-96b3-43c4-8a74-2d3536a42f67"). InnerVolumeSpecName "kube-api-access-8b4ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:12.299420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:12.299382 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-xks-ca" pod "2b11d16e-96b3-43c4-8a74-2d3536a42f67" (UID: "2b11d16e-96b3-43c4-8a74-2d3536a42f67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:12.398180 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:12.398144 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:12.398180 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:12.398179 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b4ws\" (UniqueName: \"kubernetes.io/projected/2b11d16e-96b3-43c4-8a74-2d3536a42f67-kube-api-access-8b4ws\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:12.398427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:12.398194 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b11d16e-96b3-43c4-8a74-2d3536a42f67-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:12.981757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:12.981730 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" Nov 03 20:12:12.981757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:12.981723 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5" event={"ID":"2b11d16e-96b3-43c4-8a74-2d3536a42f67","Type":"ContainerDied","Data":"0063b9c1cc815e68bf38a2e7e5509b177111b6e7e5d309f56f9192941190e50c"} Nov 03 20:12:12.982289 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:12.981785 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0063b9c1cc815e68bf38a2e7e5509b177111b6e7e5d309f56f9192941190e50c" Nov 03 20:12:13.131291 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:13.131261 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" Nov 03 20:12:13.308197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:13.308166 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/669b4131-2371-4b01-b932-7035e230d493-kube-api-access-jx4lf\" (UniqueName: \"kubernetes.io/projected/669b4131-2371-4b01-b932-7035e230d493-kube-api-access-jx4lf\") pod \"669b4131-2371-4b01-b932-7035e230d493\" (UID: \"669b4131-2371-4b01-b932-7035e230d493\") " Nov 03 20:12:13.308367 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:13.308275 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/669b4131-2371-4b01-b932-7035e230d493-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669b4131-2371-4b01-b932-7035e230d493-maas-api-openshift-ca\") pod \"669b4131-2371-4b01-b932-7035e230d493\" (UID: \"669b4131-2371-4b01-b932-7035e230d493\") " Nov 03 20:12:13.308367 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:13.308345 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/669b4131-2371-4b01-b932-7035e230d493-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669b4131-2371-4b01-b932-7035e230d493-maas-api-xks-ca\") pod \"669b4131-2371-4b01-b932-7035e230d493\" (UID: \"669b4131-2371-4b01-b932-7035e230d493\") " Nov 03 20:12:13.308701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:13.308666 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/669b4131-2371-4b01-b932-7035e230d493-maas-api-openshift-ca" pod "669b4131-2371-4b01-b932-7035e230d493" (UID: "669b4131-2371-4b01-b932-7035e230d493"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:13.310486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:13.310461 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/669b4131-2371-4b01-b932-7035e230d493-kube-api-access-jx4lf" pod "669b4131-2371-4b01-b932-7035e230d493" (UID: "669b4131-2371-4b01-b932-7035e230d493"). InnerVolumeSpecName "kube-api-access-jx4lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:13.310603 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:13.310514 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/669b4131-2371-4b01-b932-7035e230d493-maas-api-xks-ca" pod "669b4131-2371-4b01-b932-7035e230d493" (UID: "669b4131-2371-4b01-b932-7035e230d493"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:13.409710 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:13.409677 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx4lf\" (UniqueName: \"kubernetes.io/projected/669b4131-2371-4b01-b932-7035e230d493-kube-api-access-jx4lf\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:13.409710 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:13.409705 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/669b4131-2371-4b01-b932-7035e230d493-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:13.409710 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:13.409716 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/669b4131-2371-4b01-b932-7035e230d493-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:13.988986 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:13.988958 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" Nov 03 20:12:13.989361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:13.988990 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c" event={"ID":"669b4131-2371-4b01-b932-7035e230d493","Type":"ContainerDied","Data":"390f6b69836a76c0bcb00c1f27fc80e625eed6fb0ea75b1850ef4ef4da84d1e3"} Nov 03 20:12:13.989361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:13.989040 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="390f6b69836a76c0bcb00c1f27fc80e625eed6fb0ea75b1850ef4ef4da84d1e3" Nov 03 20:12:18.074492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.074456 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r"] Nov 03 20:12:18.075213 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.075191 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2b11d16e-96b3-43c4-8a74-2d3536a42f67" containerName="revoke-keys" Nov 03 20:12:18.075257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.075216 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b11d16e-96b3-43c4-8a74-2d3536a42f67" containerName="revoke-keys" Nov 03 20:12:18.075257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.075247 4457 cpu_manager.go:414] "containerMap: removing container" podUID="669b4131-2371-4b01-b932-7035e230d493" containerName="revoke-keys" Nov 03 20:12:18.075322 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.075255 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="669b4131-2371-4b01-b932-7035e230d493" containerName="revoke-keys" Nov 03 20:12:18.075401 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.075374 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b11d16e-96b3-43c4-8a74-2d3536a42f67" containerName="revoke-keys" Nov 03 20:12:18.075435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.075408 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="669b4131-2371-4b01-b932-7035e230d493" containerName="revoke-keys" Nov 03 20:12:18.079460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.079437 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" Nov 03 20:12:18.085101 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.083661 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:12:18.085101 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.084637 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:12:18.086349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.086324 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r"] Nov 03 20:12:18.155664 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.155630 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv7kz\" (UniqueName: \"kubernetes.io/projected/993aa66d-9758-47e9-b83c-9341d0e26995-kube-api-access-lv7kz\") pod \"maas-api-revoke-sub-06c5e3fd-5kp9r\" (UID: \"993aa66d-9758-47e9-b83c-9341d0e26995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" Nov 03 20:12:18.155817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.155769 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06c5e3fd-5kp9r\" (UID: \"993aa66d-9758-47e9-b83c-9341d0e26995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" Nov 03 20:12:18.155817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.155810 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06c5e3fd-5kp9r\" (UID: \"993aa66d-9758-47e9-b83c-9341d0e26995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" Nov 03 20:12:18.257453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.257416 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv7kz\" (UniqueName: \"kubernetes.io/projected/993aa66d-9758-47e9-b83c-9341d0e26995-kube-api-access-lv7kz\") pod \"maas-api-revoke-sub-06c5e3fd-5kp9r\" (UID: \"993aa66d-9758-47e9-b83c-9341d0e26995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" Nov 03 20:12:18.257611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.257529 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06c5e3fd-5kp9r\" (UID: \"993aa66d-9758-47e9-b83c-9341d0e26995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" Nov 03 20:12:18.257611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.257572 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06c5e3fd-5kp9r\" (UID: \"993aa66d-9758-47e9-b83c-9341d0e26995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" Nov 03 20:12:18.258313 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.258284 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06c5e3fd-5kp9r\" (UID: \"993aa66d-9758-47e9-b83c-9341d0e26995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" Nov 03 20:12:18.260099 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.260069 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06c5e3fd-5kp9r\" (UID: \"993aa66d-9758-47e9-b83c-9341d0e26995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" Nov 03 20:12:18.269732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.269701 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv7kz\" (UniqueName: \"kubernetes.io/projected/993aa66d-9758-47e9-b83c-9341d0e26995-kube-api-access-lv7kz\") pod \"maas-api-revoke-sub-06c5e3fd-5kp9r\" (UID: \"993aa66d-9758-47e9-b83c-9341d0e26995\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" Nov 03 20:12:18.397752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.397654 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" Nov 03 20:12:18.678534 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:12:18.678502 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod993aa66d_9758_47e9_b83c_9341d0e26995.slice/crio-e985151c78fc8312b41b758139ab2d07614ff8447f39b5c615525d1153b345a2 WatchSource:0}: Error finding container e985151c78fc8312b41b758139ab2d07614ff8447f39b5c615525d1153b345a2: Status 404 returned error can't find the container with id e985151c78fc8312b41b758139ab2d07614ff8447f39b5c615525d1153b345a2 Nov 03 20:12:18.678534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:18.678508 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r"] Nov 03 20:12:19.023117 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:19.023083 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" event={"ID":"993aa66d-9758-47e9-b83c-9341d0e26995","Type":"ContainerStarted","Data":"94b49037c061ae1e64a0f7ee9119f6291c2e9899b530cdd594cbcf10d98fff64"} Nov 03 20:12:19.023280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:19.023132 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" event={"ID":"993aa66d-9758-47e9-b83c-9341d0e26995","Type":"ContainerStarted","Data":"e985151c78fc8312b41b758139ab2d07614ff8447f39b5c615525d1153b345a2"} Nov 03 20:12:19.044824 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:19.044780 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" podStartSLOduration=1.044766913 podStartE2EDuration="1.044766913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:19.042227253 +0000 UTC m=+917.417824001" watchObservedRunningTime="2026-11-03 20:12:19.044766913 +0000 UTC m=+917.420363660" Nov 03 20:12:20.033323 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.033225 4457 generic.go:356] "Generic (PLEG): container finished" podID="993aa66d-9758-47e9-b83c-9341d0e26995" containerID="94b49037c061ae1e64a0f7ee9119f6291c2e9899b530cdd594cbcf10d98fff64" exitCode=0 Nov 03 20:12:20.033732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.033310 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" event={"ID":"993aa66d-9758-47e9-b83c-9341d0e26995","Type":"ContainerDied","Data":"94b49037c061ae1e64a0f7ee9119f6291c2e9899b530cdd594cbcf10d98fff64"} Nov 03 20:12:20.724092 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.724057 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd"] Nov 03 20:12:20.728820 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.728793 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" Nov 03 20:12:20.735374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.735346 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd"] Nov 03 20:12:20.800479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.800430 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d33091-7thdd\" (UID: \"8625493a-9238-4f7a-86cb-320fd834d9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" Nov 03 20:12:20.800647 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.800496 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d33091-7thdd\" (UID: \"8625493a-9238-4f7a-86cb-320fd834d9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" Nov 03 20:12:20.800732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.800638 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcdts\" (UniqueName: \"kubernetes.io/projected/8625493a-9238-4f7a-86cb-320fd834d9e0-kube-api-access-kcdts\") pod \"maas-api-revoke-sub-60d33091-7thdd\" (UID: \"8625493a-9238-4f7a-86cb-320fd834d9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" Nov 03 20:12:20.901885 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.901841 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d33091-7thdd\" (UID: \"8625493a-9238-4f7a-86cb-320fd834d9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" Nov 03 20:12:20.902055 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.901894 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d33091-7thdd\" (UID: \"8625493a-9238-4f7a-86cb-320fd834d9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" Nov 03 20:12:20.902055 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.901931 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcdts\" (UniqueName: \"kubernetes.io/projected/8625493a-9238-4f7a-86cb-320fd834d9e0-kube-api-access-kcdts\") pod \"maas-api-revoke-sub-60d33091-7thdd\" (UID: \"8625493a-9238-4f7a-86cb-320fd834d9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" Nov 03 20:12:20.902671 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.902645 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60d33091-7thdd\" (UID: \"8625493a-9238-4f7a-86cb-320fd834d9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" Nov 03 20:12:20.904243 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.904225 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60d33091-7thdd\" (UID: \"8625493a-9238-4f7a-86cb-320fd834d9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" Nov 03 20:12:20.913912 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:20.913884 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcdts\" (UniqueName: \"kubernetes.io/projected/8625493a-9238-4f7a-86cb-320fd834d9e0-kube-api-access-kcdts\") pod \"maas-api-revoke-sub-60d33091-7thdd\" (UID: \"8625493a-9238-4f7a-86cb-320fd834d9e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" Nov 03 20:12:21.047858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:21.047829 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" Nov 03 20:12:21.193164 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:21.193135 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" Nov 03 20:12:21.308197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:21.308170 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-openshift-ca\") pod \"993aa66d-9758-47e9-b83c-9341d0e26995\" (UID: \"993aa66d-9758-47e9-b83c-9341d0e26995\") " Nov 03 20:12:21.308371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:21.308344 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-xks-ca\") pod \"993aa66d-9758-47e9-b83c-9341d0e26995\" (UID: \"993aa66d-9758-47e9-b83c-9341d0e26995\") " Nov 03 20:12:21.308443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:21.308374 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/993aa66d-9758-47e9-b83c-9341d0e26995-kube-api-access-lv7kz\" (UniqueName: \"kubernetes.io/projected/993aa66d-9758-47e9-b83c-9341d0e26995-kube-api-access-lv7kz\") pod \"993aa66d-9758-47e9-b83c-9341d0e26995\" (UID: \"993aa66d-9758-47e9-b83c-9341d0e26995\") " Nov 03 20:12:21.308593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:21.308571 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-openshift-ca" pod "993aa66d-9758-47e9-b83c-9341d0e26995" (UID: "993aa66d-9758-47e9-b83c-9341d0e26995"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:21.308807 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:21.308787 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:21.310530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:21.310511 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/993aa66d-9758-47e9-b83c-9341d0e26995-kube-api-access-lv7kz" pod "993aa66d-9758-47e9-b83c-9341d0e26995" (UID: "993aa66d-9758-47e9-b83c-9341d0e26995"). InnerVolumeSpecName "kube-api-access-lv7kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:21.310608 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:21.310510 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-xks-ca" pod "993aa66d-9758-47e9-b83c-9341d0e26995" (UID: "993aa66d-9758-47e9-b83c-9341d0e26995"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:21.352555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:21.352531 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd"] Nov 03 20:12:21.354050 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:12:21.354025 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8625493a_9238_4f7a_86cb_320fd834d9e0.slice/crio-60e50df9322845afad907b227749a25aa58b39bc3a17cb3cfe3d16fde1bf2579 WatchSource:0}: Error finding container 60e50df9322845afad907b227749a25aa58b39bc3a17cb3cfe3d16fde1bf2579: Status 404 returned error can't find the container with id 60e50df9322845afad907b227749a25aa58b39bc3a17cb3cfe3d16fde1bf2579 Nov 03 20:12:21.410336 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:21.410285 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/993aa66d-9758-47e9-b83c-9341d0e26995-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:21.410336 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:21.410336 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv7kz\" (UniqueName: \"kubernetes.io/projected/993aa66d-9758-47e9-b83c-9341d0e26995-kube-api-access-lv7kz\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:22.047243 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:22.047208 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" Nov 03 20:12:22.047243 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:22.047226 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r" event={"ID":"993aa66d-9758-47e9-b83c-9341d0e26995","Type":"ContainerDied","Data":"e985151c78fc8312b41b758139ab2d07614ff8447f39b5c615525d1153b345a2"} Nov 03 20:12:22.047621 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:22.047284 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e985151c78fc8312b41b758139ab2d07614ff8447f39b5c615525d1153b345a2" Nov 03 20:12:22.049700 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:22.049665 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" event={"ID":"8625493a-9238-4f7a-86cb-320fd834d9e0","Type":"ContainerStarted","Data":"d44f70279df7fb1352959fe0563eccde3383166eb33ab6dd8434d4f4d0bde75d"} Nov 03 20:12:22.050089 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:22.049720 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" event={"ID":"8625493a-9238-4f7a-86cb-320fd834d9e0","Type":"ContainerStarted","Data":"60e50df9322845afad907b227749a25aa58b39bc3a17cb3cfe3d16fde1bf2579"} Nov 03 20:12:22.237347 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:22.237238 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" podStartSLOduration=2.237218593 podStartE2EDuration="2.237218593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:22.06971616 +0000 UTC m=+920.445312908" watchObservedRunningTime="2026-11-03 20:12:22.237218593 +0000 UTC m=+920.612815362" Nov 03 20:12:23.059436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:23.059372 4457 generic.go:356] "Generic (PLEG): container finished" podID="8625493a-9238-4f7a-86cb-320fd834d9e0" containerID="d44f70279df7fb1352959fe0563eccde3383166eb33ab6dd8434d4f4d0bde75d" exitCode=0 Nov 03 20:12:23.059842 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:23.059457 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" event={"ID":"8625493a-9238-4f7a-86cb-320fd834d9e0","Type":"ContainerDied","Data":"d44f70279df7fb1352959fe0563eccde3383166eb33ab6dd8434d4f4d0bde75d"} Nov 03 20:12:23.853106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:23.853071 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52"] Nov 03 20:12:23.853585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:23.853572 4457 cpu_manager.go:414] "containerMap: removing container" podUID="993aa66d-9758-47e9-b83c-9341d0e26995" containerName="revoke-keys" Nov 03 20:12:23.853637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:23.853587 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="993aa66d-9758-47e9-b83c-9341d0e26995" containerName="revoke-keys" Nov 03 20:12:23.853686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:23.853677 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="993aa66d-9758-47e9-b83c-9341d0e26995" containerName="revoke-keys" Nov 03 20:12:23.857186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:23.857169 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" Nov 03 20:12:23.869925 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:23.869897 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52"] Nov 03 20:12:23.936954 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:23.936914 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e563a98-h9k52\" (UID: \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" Nov 03 20:12:23.937120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:23.937062 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4qr\" (UniqueName: \"kubernetes.io/projected/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-kube-api-access-4f4qr\") pod \"maas-api-revoke-sub-2e563a98-h9k52\" (UID: \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" Nov 03 20:12:23.937185 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:23.937170 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e563a98-h9k52\" (UID: \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" Nov 03 20:12:24.038895 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.038861 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e563a98-h9k52\" (UID: \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" Nov 03 20:12:24.039076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.038969 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4qr\" (UniqueName: \"kubernetes.io/projected/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-kube-api-access-4f4qr\") pod \"maas-api-revoke-sub-2e563a98-h9k52\" (UID: \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" Nov 03 20:12:24.039076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.039065 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e563a98-h9k52\" (UID: \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" Nov 03 20:12:24.039734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.039706 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e563a98-h9k52\" (UID: \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" Nov 03 20:12:24.041269 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.041246 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e563a98-h9k52\" (UID: \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" Nov 03 20:12:24.053583 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.053560 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4qr\" (UniqueName: \"kubernetes.io/projected/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-kube-api-access-4f4qr\") pod \"maas-api-revoke-sub-2e563a98-h9k52\" (UID: \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" Nov 03 20:12:24.178713 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.178660 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" Nov 03 20:12:24.237273 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.236530 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" Nov 03 20:12:24.342629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.342590 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-xks-ca\") pod \"8625493a-9238-4f7a-86cb-320fd834d9e0\" (UID: \"8625493a-9238-4f7a-86cb-320fd834d9e0\") " Nov 03 20:12:24.342908 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.342659 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-openshift-ca\") pod \"8625493a-9238-4f7a-86cb-320fd834d9e0\" (UID: \"8625493a-9238-4f7a-86cb-320fd834d9e0\") " Nov 03 20:12:24.342908 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.342824 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8625493a-9238-4f7a-86cb-320fd834d9e0-kube-api-access-kcdts\" (UniqueName: \"kubernetes.io/projected/8625493a-9238-4f7a-86cb-320fd834d9e0-kube-api-access-kcdts\") pod \"8625493a-9238-4f7a-86cb-320fd834d9e0\" (UID: \"8625493a-9238-4f7a-86cb-320fd834d9e0\") " Nov 03 20:12:24.343228 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.343201 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-openshift-ca" pod "8625493a-9238-4f7a-86cb-320fd834d9e0" (UID: "8625493a-9238-4f7a-86cb-320fd834d9e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:24.345044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.345020 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-xks-ca" pod "8625493a-9238-4f7a-86cb-320fd834d9e0" (UID: "8625493a-9238-4f7a-86cb-320fd834d9e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:24.345339 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.345319 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8625493a-9238-4f7a-86cb-320fd834d9e0-kube-api-access-kcdts" pod "8625493a-9238-4f7a-86cb-320fd834d9e0" (UID: "8625493a-9238-4f7a-86cb-320fd834d9e0"). InnerVolumeSpecName "kube-api-access-kcdts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:24.444460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.444371 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcdts\" (UniqueName: \"kubernetes.io/projected/8625493a-9238-4f7a-86cb-320fd834d9e0-kube-api-access-kcdts\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:24.444460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.444410 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:24.444460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.444421 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8625493a-9238-4f7a-86cb-320fd834d9e0-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:24.481481 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.479709 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52"] Nov 03 20:12:24.481481 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:12:24.481454 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod536d878c_d7a4_43ba_bcbb_65ab24fb30c4.slice/crio-88e5039caa2541381536e6284984440e6da49451fea7855e20d51c180582011f WatchSource:0}: Error finding container 88e5039caa2541381536e6284984440e6da49451fea7855e20d51c180582011f: Status 404 returned error can't find the container with id 88e5039caa2541381536e6284984440e6da49451fea7855e20d51c180582011f Nov 03 20:12:24.912116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.912070 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf"] Nov 03 20:12:24.912582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.912567 4457 cpu_manager.go:414] "containerMap: removing container" podUID="8625493a-9238-4f7a-86cb-320fd834d9e0" containerName="revoke-keys" Nov 03 20:12:24.912648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.912583 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8625493a-9238-4f7a-86cb-320fd834d9e0" containerName="revoke-keys" Nov 03 20:12:24.912684 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.912681 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="8625493a-9238-4f7a-86cb-320fd834d9e0" containerName="revoke-keys" Nov 03 20:12:24.915921 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.915905 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" Nov 03 20:12:24.968425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:24.961376 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf"] Nov 03 20:12:25.050744 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.050700 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-285d4478-zd4rf\" (UID: \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" Nov 03 20:12:25.050890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.050851 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-285d4478-zd4rf\" (UID: \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" Nov 03 20:12:25.050933 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.050890 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx999\" (UniqueName: \"kubernetes.io/projected/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-kube-api-access-nx999\") pod \"maas-api-revoke-sub-285d4478-zd4rf\" (UID: \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" Nov 03 20:12:25.075234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.075206 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" Nov 03 20:12:25.075426 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.075230 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd" event={"ID":"8625493a-9238-4f7a-86cb-320fd834d9e0","Type":"ContainerDied","Data":"60e50df9322845afad907b227749a25aa58b39bc3a17cb3cfe3d16fde1bf2579"} Nov 03 20:12:25.075426 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.075287 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60e50df9322845afad907b227749a25aa58b39bc3a17cb3cfe3d16fde1bf2579" Nov 03 20:12:25.077149 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.077081 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" event={"ID":"536d878c-d7a4-43ba-bcbb-65ab24fb30c4","Type":"ContainerStarted","Data":"8e81450a683a2e8a52d3ee2e70665816879ea8968b7a2b395beca34360a256e7"} Nov 03 20:12:25.077287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.077175 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" event={"ID":"536d878c-d7a4-43ba-bcbb-65ab24fb30c4","Type":"ContainerStarted","Data":"88e5039caa2541381536e6284984440e6da49451fea7855e20d51c180582011f"} Nov 03 20:12:25.103342 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.103300 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" podStartSLOduration=2.10328709 podStartE2EDuration="2.10328709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:25.100767479 +0000 UTC m=+923.476364227" watchObservedRunningTime="2026-11-03 20:12:25.10328709 +0000 UTC m=+923.478883853" Nov 03 20:12:25.152493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.152458 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-285d4478-zd4rf\" (UID: \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" Nov 03 20:12:25.152701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.152509 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx999\" (UniqueName: \"kubernetes.io/projected/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-kube-api-access-nx999\") pod \"maas-api-revoke-sub-285d4478-zd4rf\" (UID: \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" Nov 03 20:12:25.152701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.152584 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-285d4478-zd4rf\" (UID: \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" Nov 03 20:12:25.153229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.153198 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-285d4478-zd4rf\" (UID: \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" Nov 03 20:12:25.154903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.154875 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-285d4478-zd4rf\" (UID: \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" Nov 03 20:12:25.165181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.165160 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx999\" (UniqueName: \"kubernetes.io/projected/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-kube-api-access-nx999\") pod \"maas-api-revoke-sub-285d4478-zd4rf\" (UID: \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" Nov 03 20:12:25.229769 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.229737 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" Nov 03 20:12:25.512699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:25.512664 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf"] Nov 03 20:12:25.516043 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:12:25.516019 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2fe4f47_0d4f_4fd8_9a7f_c61225c80cb1.slice/crio-e0d44f582abac5c3bc1df6c6a29ca5c2a45d6d49da4264e0f78a1b10d0ab02e0 WatchSource:0}: Error finding container e0d44f582abac5c3bc1df6c6a29ca5c2a45d6d49da4264e0f78a1b10d0ab02e0: Status 404 returned error can't find the container with id e0d44f582abac5c3bc1df6c6a29ca5c2a45d6d49da4264e0f78a1b10d0ab02e0 Nov 03 20:12:26.084787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:26.084755 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" event={"ID":"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1","Type":"ContainerStarted","Data":"7c6187dec557345a8b20b75deb079bdc1f05a0468880bc4d503abc98c1a70599"} Nov 03 20:12:26.084974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:26.084809 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" event={"ID":"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1","Type":"ContainerStarted","Data":"e0d44f582abac5c3bc1df6c6a29ca5c2a45d6d49da4264e0f78a1b10d0ab02e0"} Nov 03 20:12:26.096625 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:26.096589 4457 generic.go:356] "Generic (PLEG): container finished" podID="536d878c-d7a4-43ba-bcbb-65ab24fb30c4" containerID="8e81450a683a2e8a52d3ee2e70665816879ea8968b7a2b395beca34360a256e7" exitCode=0 Nov 03 20:12:26.096788 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:26.096649 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" event={"ID":"536d878c-d7a4-43ba-bcbb-65ab24fb30c4","Type":"ContainerDied","Data":"8e81450a683a2e8a52d3ee2e70665816879ea8968b7a2b395beca34360a256e7"} Nov 03 20:12:26.105765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:26.105723 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" podStartSLOduration=2.10571171 podStartE2EDuration="2.10571171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:26.103983198 +0000 UTC m=+924.479579942" watchObservedRunningTime="2026-11-03 20:12:26.10571171 +0000 UTC m=+924.481308468" Nov 03 20:12:27.106839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:27.106803 4457 generic.go:356] "Generic (PLEG): container finished" podID="c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1" containerID="7c6187dec557345a8b20b75deb079bdc1f05a0468880bc4d503abc98c1a70599" exitCode=0 Nov 03 20:12:27.107255 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:27.106894 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" event={"ID":"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1","Type":"ContainerDied","Data":"7c6187dec557345a8b20b75deb079bdc1f05a0468880bc4d503abc98c1a70599"} Nov 03 20:12:27.279816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:27.279790 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" Nov 03 20:12:27.379456 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:27.379319 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-xks-ca\") pod \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\" (UID: \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\") " Nov 03 20:12:27.379639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:27.379524 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-kube-api-access-4f4qr\" (UniqueName: \"kubernetes.io/projected/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-kube-api-access-4f4qr\") pod \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\" (UID: \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\") " Nov 03 20:12:27.379639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:27.379557 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-openshift-ca\") pod \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\" (UID: \"536d878c-d7a4-43ba-bcbb-65ab24fb30c4\") " Nov 03 20:12:27.380302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:27.380272 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-openshift-ca" pod "536d878c-d7a4-43ba-bcbb-65ab24fb30c4" (UID: "536d878c-d7a4-43ba-bcbb-65ab24fb30c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:27.381798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:27.381757 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-xks-ca" pod "536d878c-d7a4-43ba-bcbb-65ab24fb30c4" (UID: "536d878c-d7a4-43ba-bcbb-65ab24fb30c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:27.382305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:27.382282 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-kube-api-access-4f4qr" pod "536d878c-d7a4-43ba-bcbb-65ab24fb30c4" (UID: "536d878c-d7a4-43ba-bcbb-65ab24fb30c4"). InnerVolumeSpecName "kube-api-access-4f4qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:27.480869 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:27.480815 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:27.480869 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:27.480870 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f4qr\" (UniqueName: \"kubernetes.io/projected/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-kube-api-access-4f4qr\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:27.481078 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:27.480887 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/536d878c-d7a4-43ba-bcbb-65ab24fb30c4-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:28.115724 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:28.115689 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" event={"ID":"536d878c-d7a4-43ba-bcbb-65ab24fb30c4","Type":"ContainerDied","Data":"88e5039caa2541381536e6284984440e6da49451fea7855e20d51c180582011f"} Nov 03 20:12:28.115724 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:28.115719 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52" Nov 03 20:12:28.116233 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:28.115745 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88e5039caa2541381536e6284984440e6da49451fea7855e20d51c180582011f" Nov 03 20:12:28.277100 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:28.277071 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" Nov 03 20:12:28.391961 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:28.391871 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-xks-ca\") pod \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\" (UID: \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\") " Nov 03 20:12:28.392112 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:28.391986 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-kube-api-access-nx999\" (UniqueName: \"kubernetes.io/projected/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-kube-api-access-nx999\") pod \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\" (UID: \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\") " Nov 03 20:12:28.392112 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:28.392046 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-openshift-ca\") pod \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\" (UID: \"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1\") " Nov 03 20:12:28.392624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:28.392596 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-openshift-ca" pod "c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1" (UID: "c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:28.394211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:28.394179 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-xks-ca" pod "c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1" (UID: "c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:28.394341 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:28.394277 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-kube-api-access-nx999" pod "c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1" (UID: "c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1"). InnerVolumeSpecName "kube-api-access-nx999". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:28.493969 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:28.493926 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:28.493969 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:28.493957 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx999\" (UniqueName: \"kubernetes.io/projected/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-kube-api-access-nx999\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:28.493969 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:28.493969 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:29.124458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:29.124427 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" Nov 03 20:12:29.124841 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:29.124471 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf" event={"ID":"c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1","Type":"ContainerDied","Data":"e0d44f582abac5c3bc1df6c6a29ca5c2a45d6d49da4264e0f78a1b10d0ab02e0"} Nov 03 20:12:29.124841 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:29.124525 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0d44f582abac5c3bc1df6c6a29ca5c2a45d6d49da4264e0f78a1b10d0ab02e0" Nov 03 20:12:30.762412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.758292 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf"] Nov 03 20:12:30.762412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.758999 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1" containerName="revoke-keys" Nov 03 20:12:30.762412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.759021 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1" containerName="revoke-keys" Nov 03 20:12:30.762412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.759084 4457 cpu_manager.go:414] "containerMap: removing container" podUID="536d878c-d7a4-43ba-bcbb-65ab24fb30c4" containerName="revoke-keys" Nov 03 20:12:30.762412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.759092 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536d878c-d7a4-43ba-bcbb-65ab24fb30c4" containerName="revoke-keys" Nov 03 20:12:30.762412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.759211 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="536d878c-d7a4-43ba-bcbb-65ab24fb30c4" containerName="revoke-keys" Nov 03 20:12:30.762412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.759225 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1" containerName="revoke-keys" Nov 03 20:12:30.779084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.777600 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:30.779084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.778602 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf"] Nov 03 20:12:30.785411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.782334 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-3aba31-openshift-default-dockercfg-s4h77\"" Nov 03 20:12:30.947675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.947637 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:30.947964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.947931 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6kcv\" (UniqueName: \"kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-kube-api-access-f6kcv\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:30.948374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.948134 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:30.948782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.948761 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:30.949002 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.948981 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:30.949231 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.949213 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:30.949450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.949430 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:30.949639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.949622 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:30.949815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:30.949795 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.050996 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.051049 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6kcv\" (UniqueName: \"kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-kube-api-access-f6kcv\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.051095 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.051134 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.051188 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.051247 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.051290 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.051323 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.051350 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.052461 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.053042 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.053739 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.053970 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.055411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.054171 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.062680 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.061320 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.063135 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.063087 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.070740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.070694 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.089083 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.089029 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6kcv\" (UniqueName: \"kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-kube-api-access-f6kcv\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.137852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.137594 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:31.263224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.263194 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw"] Nov 03 20:12:31.369459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.354495 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw"] Nov 03 20:12:31.369459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.354652 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.369459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.360232 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a6f98b\"/\"distinct-w5-a6f98b-kserve-self-signed-certs\"" Nov 03 20:12:31.571415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.545943 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-home\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.571415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.546067 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-model-cache\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.571415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.546101 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tmp-dir\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.571415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.546127 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tls-certs\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.571415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.546163 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-dshm\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.571415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.546193 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8cr\" (UniqueName: \"kubernetes.io/projected/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-kube-api-access-7v8cr\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.661072 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.660983 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8cr\" (UniqueName: \"kubernetes.io/projected/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-kube-api-access-7v8cr\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.662547 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.662517 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-home\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.663166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.663147 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-model-cache\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.669664 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.669643 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tmp-dir\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.670171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.670153 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tls-certs\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.730495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.730453 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-dshm\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.730689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.670103 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tmp-dir\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.730689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.669592 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-model-cache\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.730689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.663013 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-home\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.744852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.743421 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8cr\" (UniqueName: \"kubernetes.io/projected/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-kube-api-access-7v8cr\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.856673 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.856607 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tls-certs\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.861647 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.861585 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-dshm\") pod \"distinct-w5-a6f98b-kserve-9c7556976-jzgtw\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.972443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.971838 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:31.997422 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:31.994594 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf"] Nov 03 20:12:32.049836 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:12:32.049808 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f3184c_7322_4684_94e1_9e5e4eecd6e7.slice/crio-d0ff136a0c4617f7c6ebdfa46eaf2c781020c5d3ad07e39bdc28a270121b2029 WatchSource:0}: Error finding container d0ff136a0c4617f7c6ebdfa46eaf2c781020c5d3ad07e39bdc28a270121b2029: Status 404 returned error can't find the container with id d0ff136a0c4617f7c6ebdfa46eaf2c781020c5d3ad07e39bdc28a270121b2029 Nov 03 20:12:32.053651 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:32.053611 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:32.053740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:32.053685 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:32.053740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:32.053713 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:32.609413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:32.604657 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" event={"ID":"97f3184c-7322-4684-94e1-9e5e4eecd6e7","Type":"ContainerStarted","Data":"d0ff136a0c4617f7c6ebdfa46eaf2c781020c5d3ad07e39bdc28a270121b2029"} Nov 03 20:12:32.728502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:32.727064 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw"] Nov 03 20:12:32.728502 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:12:32.727070 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc209ddd2_c195_4bd5_ba52_b6f13b6a35a1.slice/crio-037097b554fa09f844716c78f975f1a7e5b8657e31b38eae5afaa4fb85a326cd WatchSource:0}: Error finding container 037097b554fa09f844716c78f975f1a7e5b8657e31b38eae5afaa4fb85a326cd: Status 404 returned error can't find the container with id 037097b554fa09f844716c78f975f1a7e5b8657e31b38eae5afaa4fb85a326cd Nov 03 20:12:33.614486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.614446 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" event={"ID":"97f3184c-7322-4684-94e1-9e5e4eecd6e7","Type":"ContainerStarted","Data":"de1a8c74891e1f40121ffa0dbba05bc979f4d253660c1b97c7bfcd3b45a80b2c"} Nov 03 20:12:33.616513 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.616474 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" event={"ID":"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1","Type":"ContainerStarted","Data":"a074aca130d8f183d18598261425a6c1295add2b1f78eec7bdefb3b184933c8d"} Nov 03 20:12:33.616513 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.616515 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" event={"ID":"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1","Type":"ContainerStarted","Data":"037097b554fa09f844716c78f975f1a7e5b8657e31b38eae5afaa4fb85a326cd"} Nov 03 20:12:33.641174 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.641121 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" podStartSLOduration=3.641106738 podStartE2EDuration="3.641106738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:33.637943927 +0000 UTC m=+932.013540675" watchObservedRunningTime="2026-11-03 20:12:33.641106738 +0000 UTC m=+932.016703485" Nov 03 20:12:33.686437 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.685145 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podStartSLOduration=2.6851234059999998 podStartE2EDuration="2.685123406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:33.671683053 +0000 UTC m=+932.047279802" watchObservedRunningTime="2026-11-03 20:12:33.685123406 +0000 UTC m=+932.060720158" Nov 03 20:12:33.690406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.689327 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr"] Nov 03 20:12:33.694161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.694104 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" Nov 03 20:12:33.699920 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.699890 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr"] Nov 03 20:12:33.703177 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.703142 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:12:33.703433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.703381 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:12:33.716454 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.716356 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwccl\" (UniqueName: \"kubernetes.io/projected/98df7398-6420-4228-b013-94e01b98bccc-kube-api-access-vwccl\") pod \"maas-api-revoke-sub-949440a8-wtmdr\" (UID: \"98df7398-6420-4228-b013-94e01b98bccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" Nov 03 20:12:33.716585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.716536 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98df7398-6420-4228-b013-94e01b98bccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-949440a8-wtmdr\" (UID: \"98df7398-6420-4228-b013-94e01b98bccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" Nov 03 20:12:33.716647 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.716584 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98df7398-6420-4228-b013-94e01b98bccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-949440a8-wtmdr\" (UID: \"98df7398-6420-4228-b013-94e01b98bccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" Nov 03 20:12:33.817450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.817372 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98df7398-6420-4228-b013-94e01b98bccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-949440a8-wtmdr\" (UID: \"98df7398-6420-4228-b013-94e01b98bccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" Nov 03 20:12:33.817660 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.817465 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98df7398-6420-4228-b013-94e01b98bccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-949440a8-wtmdr\" (UID: \"98df7398-6420-4228-b013-94e01b98bccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" Nov 03 20:12:33.817795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.817729 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwccl\" (UniqueName: \"kubernetes.io/projected/98df7398-6420-4228-b013-94e01b98bccc-kube-api-access-vwccl\") pod \"maas-api-revoke-sub-949440a8-wtmdr\" (UID: \"98df7398-6420-4228-b013-94e01b98bccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" Nov 03 20:12:33.819026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.819000 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98df7398-6420-4228-b013-94e01b98bccc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-949440a8-wtmdr\" (UID: \"98df7398-6420-4228-b013-94e01b98bccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" Nov 03 20:12:33.820789 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.820771 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98df7398-6420-4228-b013-94e01b98bccc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-949440a8-wtmdr\" (UID: \"98df7398-6420-4228-b013-94e01b98bccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" Nov 03 20:12:33.831432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.831405 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwccl\" (UniqueName: \"kubernetes.io/projected/98df7398-6420-4228-b013-94e01b98bccc-kube-api-access-vwccl\") pod \"maas-api-revoke-sub-949440a8-wtmdr\" (UID: \"98df7398-6420-4228-b013-94e01b98bccc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" Nov 03 20:12:33.982614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:33.982513 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:34.015945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:34.015913 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" Nov 03 20:12:34.139640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:34.139377 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:34.141044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:34.140793 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:12:34.141044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:34.140837 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" podUID="97f3184c-7322-4684-94e1-9e5e4eecd6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 20:12:34.328613 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:12:34.328485 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98df7398_6420_4228_b013_94e01b98bccc.slice/crio-56758f0ac4ee06255c5a03e4d2bce74e955832e479f09e3975b40ceae7b2eaca WatchSource:0}: Error finding container 56758f0ac4ee06255c5a03e4d2bce74e955832e479f09e3975b40ceae7b2eaca: Status 404 returned error can't find the container with id 56758f0ac4ee06255c5a03e4d2bce74e955832e479f09e3975b40ceae7b2eaca Nov 03 20:12:34.329188 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:34.328983 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr"] Nov 03 20:12:34.626156 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:34.626061 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" event={"ID":"98df7398-6420-4228-b013-94e01b98bccc","Type":"ContainerStarted","Data":"56758f0ac4ee06255c5a03e4d2bce74e955832e479f09e3975b40ceae7b2eaca"} Nov 03 20:12:35.139813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:35.139774 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:12:35.139970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:35.139845 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" podUID="97f3184c-7322-4684-94e1-9e5e4eecd6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 20:12:35.637373 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:35.637336 4457 generic.go:356] "Generic (PLEG): container finished" podID="98df7398-6420-4228-b013-94e01b98bccc" containerID="b9fbc7058e2d7ef4c99fdb4e43550f304ba51f1f24abed0c675465d49e4e0107" exitCode=0 Nov 03 20:12:35.637879 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:35.637424 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" event={"ID":"98df7398-6420-4228-b013-94e01b98bccc","Type":"ContainerDied","Data":"b9fbc7058e2d7ef4c99fdb4e43550f304ba51f1f24abed0c675465d49e4e0107"} Nov 03 20:12:36.139328 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.139291 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:12:36.139531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.139357 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" podUID="97f3184c-7322-4684-94e1-9e5e4eecd6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 20:12:36.149329 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:12:36.149298 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:12:36.397462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.396886 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5"] Nov 03 20:12:36.410329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.408701 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" Nov 03 20:12:36.418488 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.418074 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5"] Nov 03 20:12:36.443882 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.443014 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6rzh\" (UniqueName: \"kubernetes.io/projected/9196fff0-b36c-4ff0-a620-fb1e586b6caf-kube-api-access-d6rzh\") pod \"maas-api-revoke-sub-f7f02c74-kbfq5\" (UID: \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" Nov 03 20:12:36.443882 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.443061 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7f02c74-kbfq5\" (UID: \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" Nov 03 20:12:36.443882 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.443083 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7f02c74-kbfq5\" (UID: \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" Nov 03 20:12:36.545692 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.544105 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6rzh\" (UniqueName: \"kubernetes.io/projected/9196fff0-b36c-4ff0-a620-fb1e586b6caf-kube-api-access-d6rzh\") pod \"maas-api-revoke-sub-f7f02c74-kbfq5\" (UID: \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" Nov 03 20:12:36.545692 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.544533 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7f02c74-kbfq5\" (UID: \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" Nov 03 20:12:36.545692 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.544584 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7f02c74-kbfq5\" (UID: \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" Nov 03 20:12:36.545692 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.545546 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7f02c74-kbfq5\" (UID: \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" Nov 03 20:12:36.591994 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.568205 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7f02c74-kbfq5\" (UID: \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" Nov 03 20:12:36.610416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.607064 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6rzh\" (UniqueName: \"kubernetes.io/projected/9196fff0-b36c-4ff0-a620-fb1e586b6caf-kube-api-access-d6rzh\") pod \"maas-api-revoke-sub-f7f02c74-kbfq5\" (UID: \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" Nov 03 20:12:36.728700 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.728362 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" Nov 03 20:12:36.916869 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.916559 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" Nov 03 20:12:36.949302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.948586 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98df7398-6420-4228-b013-94e01b98bccc-kube-api-access-vwccl\" (UniqueName: \"kubernetes.io/projected/98df7398-6420-4228-b013-94e01b98bccc-kube-api-access-vwccl\") pod \"98df7398-6420-4228-b013-94e01b98bccc\" (UID: \"98df7398-6420-4228-b013-94e01b98bccc\") " Nov 03 20:12:36.949302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.948664 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98df7398-6420-4228-b013-94e01b98bccc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98df7398-6420-4228-b013-94e01b98bccc-maas-api-xks-ca\") pod \"98df7398-6420-4228-b013-94e01b98bccc\" (UID: \"98df7398-6420-4228-b013-94e01b98bccc\") " Nov 03 20:12:36.949302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.948697 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98df7398-6420-4228-b013-94e01b98bccc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98df7398-6420-4228-b013-94e01b98bccc-maas-api-openshift-ca\") pod \"98df7398-6420-4228-b013-94e01b98bccc\" (UID: \"98df7398-6420-4228-b013-94e01b98bccc\") " Nov 03 20:12:36.950154 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.949802 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98df7398-6420-4228-b013-94e01b98bccc-maas-api-openshift-ca" pod "98df7398-6420-4228-b013-94e01b98bccc" (UID: "98df7398-6420-4228-b013-94e01b98bccc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:36.952589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.952547 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98df7398-6420-4228-b013-94e01b98bccc-kube-api-access-vwccl" pod "98df7398-6420-4228-b013-94e01b98bccc" (UID: "98df7398-6420-4228-b013-94e01b98bccc"). InnerVolumeSpecName "kube-api-access-vwccl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:36.953289 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:36.953266 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98df7398-6420-4228-b013-94e01b98bccc-maas-api-xks-ca" pod "98df7398-6420-4228-b013-94e01b98bccc" (UID: "98df7398-6420-4228-b013-94e01b98bccc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:37.049814 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:37.049772 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwccl\" (UniqueName: \"kubernetes.io/projected/98df7398-6420-4228-b013-94e01b98bccc-kube-api-access-vwccl\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:37.049814 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:37.049810 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98df7398-6420-4228-b013-94e01b98bccc-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:37.049814 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:37.049820 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98df7398-6420-4228-b013-94e01b98bccc-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:37.093714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:37.093630 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5"] Nov 03 20:12:37.093714 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:12:37.093673 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9196fff0_b36c_4ff0_a620_fb1e586b6caf.slice/crio-613dd26d7590cb8966995ed798c9a7c91471574f2ce6c57a96674a7cfd0aa8de WatchSource:0}: Error finding container 613dd26d7590cb8966995ed798c9a7c91471574f2ce6c57a96674a7cfd0aa8de: Status 404 returned error can't find the container with id 613dd26d7590cb8966995ed798c9a7c91471574f2ce6c57a96674a7cfd0aa8de Nov 03 20:12:37.139440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:37.139415 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:12:37.139552 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:37.139466 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" podUID="97f3184c-7322-4684-94e1-9e5e4eecd6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 20:12:37.691411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:37.690745 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" event={"ID":"9196fff0-b36c-4ff0-a620-fb1e586b6caf","Type":"ContainerStarted","Data":"ffa6fbf13c9c783ecc3107350c88f2bf5b85f4792a6708eff390c3c642159e88"} Nov 03 20:12:37.691411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:37.690836 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" event={"ID":"9196fff0-b36c-4ff0-a620-fb1e586b6caf","Type":"ContainerStarted","Data":"613dd26d7590cb8966995ed798c9a7c91471574f2ce6c57a96674a7cfd0aa8de"} Nov 03 20:12:37.693152 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:37.693126 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" event={"ID":"98df7398-6420-4228-b013-94e01b98bccc","Type":"ContainerDied","Data":"56758f0ac4ee06255c5a03e4d2bce74e955832e479f09e3975b40ceae7b2eaca"} Nov 03 20:12:37.693263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:37.693166 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56758f0ac4ee06255c5a03e4d2bce74e955832e479f09e3975b40ceae7b2eaca" Nov 03 20:12:37.693263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:37.693223 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr" Nov 03 20:12:37.717321 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:37.717265 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" podStartSLOduration=1.717248404 podStartE2EDuration="1.717248404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:37.715591705 +0000 UTC m=+936.091188457" watchObservedRunningTime="2026-11-03 20:12:37.717248404 +0000 UTC m=+936.092845156" Nov 03 20:12:38.139187 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:38.139152 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:12:38.139615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:38.139210 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" podUID="97f3184c-7322-4684-94e1-9e5e4eecd6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 20:12:38.703773 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:38.703719 4457 generic.go:356] "Generic (PLEG): container finished" podID="9196fff0-b36c-4ff0-a620-fb1e586b6caf" containerID="ffa6fbf13c9c783ecc3107350c88f2bf5b85f4792a6708eff390c3c642159e88" exitCode=0 Nov 03 20:12:38.703984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:38.703861 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" event={"ID":"9196fff0-b36c-4ff0-a620-fb1e586b6caf","Type":"ContainerDied","Data":"ffa6fbf13c9c783ecc3107350c88f2bf5b85f4792a6708eff390c3c642159e88"} Nov 03 20:12:39.139123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:39.139082 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Nov 03 20:12:39.139305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:39.139149 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" podUID="97f3184c-7322-4684-94e1-9e5e4eecd6e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Nov 03 20:12:39.880461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:39.877410 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" Nov 03 20:12:39.922630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:39.922601 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-openshift-ca\") pod \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\" (UID: \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\") " Nov 03 20:12:39.922813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:39.922714 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-xks-ca\") pod \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\" (UID: \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\") " Nov 03 20:12:39.922907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:39.922884 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9196fff0-b36c-4ff0-a620-fb1e586b6caf-kube-api-access-d6rzh\" (UniqueName: \"kubernetes.io/projected/9196fff0-b36c-4ff0-a620-fb1e586b6caf-kube-api-access-d6rzh\") pod \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\" (UID: \"9196fff0-b36c-4ff0-a620-fb1e586b6caf\") " Nov 03 20:12:39.923126 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:39.922957 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-openshift-ca" pod "9196fff0-b36c-4ff0-a620-fb1e586b6caf" (UID: "9196fff0-b36c-4ff0-a620-fb1e586b6caf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:39.923366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:39.923339 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:39.925061 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:39.925033 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9196fff0-b36c-4ff0-a620-fb1e586b6caf-kube-api-access-d6rzh" pod "9196fff0-b36c-4ff0-a620-fb1e586b6caf" (UID: "9196fff0-b36c-4ff0-a620-fb1e586b6caf"). InnerVolumeSpecName "kube-api-access-d6rzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:39.925381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:39.925361 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-xks-ca" pod "9196fff0-b36c-4ff0-a620-fb1e586b6caf" (UID: "9196fff0-b36c-4ff0-a620-fb1e586b6caf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:40.024469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:40.024373 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9196fff0-b36c-4ff0-a620-fb1e586b6caf-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:40.024469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:40.024418 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6rzh\" (UniqueName: \"kubernetes.io/projected/9196fff0-b36c-4ff0-a620-fb1e586b6caf-kube-api-access-d6rzh\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:40.152093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:40.152060 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:40.152719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:40.152697 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:40.153707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:40.153681 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:12:40.721788 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:40.721752 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" Nov 03 20:12:40.721966 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:40.721826 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5" event={"ID":"9196fff0-b36c-4ff0-a620-fb1e586b6caf","Type":"ContainerDied","Data":"613dd26d7590cb8966995ed798c9a7c91471574f2ce6c57a96674a7cfd0aa8de"} Nov 03 20:12:40.721966 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:40.721873 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="613dd26d7590cb8966995ed798c9a7c91471574f2ce6c57a96674a7cfd0aa8de" Nov 03 20:12:41.980624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:41.980594 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:41.994116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:41.994088 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:42.749884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:42.749849 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:12:45.856661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.856569 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh"] Nov 03 20:12:45.857676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.857285 4457 cpu_manager.go:414] "containerMap: removing container" podUID="9196fff0-b36c-4ff0-a620-fb1e586b6caf" containerName="revoke-keys" Nov 03 20:12:45.857676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.857305 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9196fff0-b36c-4ff0-a620-fb1e586b6caf" containerName="revoke-keys" Nov 03 20:12:45.857676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.857361 4457 cpu_manager.go:414] "containerMap: removing container" podUID="98df7398-6420-4228-b013-94e01b98bccc" containerName="revoke-keys" Nov 03 20:12:45.857676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.857370 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98df7398-6420-4228-b013-94e01b98bccc" containerName="revoke-keys" Nov 03 20:12:45.857676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.857540 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="98df7398-6420-4228-b013-94e01b98bccc" containerName="revoke-keys" Nov 03 20:12:45.857676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.857556 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="9196fff0-b36c-4ff0-a620-fb1e586b6caf" containerName="revoke-keys" Nov 03 20:12:45.864578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.864555 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:45.869841 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.869819 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8013d9-openshift-default-dockercfg-f7g58\"" Nov 03 20:12:45.873628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.873606 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh"] Nov 03 20:12:45.927736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.927704 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:45.927887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.927759 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:45.927887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.927846 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:45.927887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.927875 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:45.928003 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.927951 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znlrn\" (UniqueName: \"kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-kube-api-access-znlrn\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:45.928052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.928036 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:45.928092 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.928085 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:45.928137 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.928104 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:45.928177 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:45.928150 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.029742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.029705 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.029918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.029770 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.029918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.029789 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.029918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.029816 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.029918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.029840 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.029918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.029885 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.030579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.029929 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.030579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.029958 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.030579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.030013 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znlrn\" (UniqueName: \"kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-kube-api-access-znlrn\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.030579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.030336 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.030579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.030443 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.030808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.030657 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.030808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.030778 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.030948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.030930 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.032714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.032691 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.032970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.032941 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.045182 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.045157 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.046214 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.046192 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znlrn\" (UniqueName: \"kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-kube-api-access-znlrn\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.105738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.105702 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw"] Nov 03 20:12:46.113618 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.113543 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.120441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.120412 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw"] Nov 03 20:12:46.171955 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.171922 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-a6f98b-openshift-default-dockercfg-xrwzw\"" Nov 03 20:12:46.183944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.183902 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:46.232053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.232024 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jd9\" (UniqueName: \"kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-kube-api-access-q6jd9\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.232195 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.232061 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.232195 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.232097 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.232264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.232207 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.232264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.232228 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.232264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.232251 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.232421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.232397 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.232462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.232450 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.232495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.232480 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27621a94-aeca-461d-bf48-cddfb8c0ef53-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.333625 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.333593 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.333850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.333636 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27621a94-aeca-461d-bf48-cddfb8c0ef53-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.333850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.333687 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jd9\" (UniqueName: \"kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-kube-api-access-q6jd9\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.333850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.333714 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.333850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.333762 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.333850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.333795 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.333850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.333819 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.334191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.333856 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.334191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.333989 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.334326 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.334299 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.334326 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.334318 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.334563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.334518 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27621a94-aeca-461d-bf48-cddfb8c0ef53-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.334692 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.334621 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.334692 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.334638 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.336284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.336259 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.336443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.336423 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.345672 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.345648 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.345756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.345657 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jd9\" (UniqueName: \"kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-kube-api-access-q6jd9\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.432148 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.432075 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:46.477946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.477912 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh"] Nov 03 20:12:46.478514 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:12:46.478485 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fd7838_627a_4d63_a0ba_ffbbe3b2afb0.slice/crio-8e62cd7cfb13aaabbd2ad3a0abe9742be5c39a8ea8b68feb81f27a2187c6556a WatchSource:0}: Error finding container 8e62cd7cfb13aaabbd2ad3a0abe9742be5c39a8ea8b68feb81f27a2187c6556a: Status 404 returned error can't find the container with id 8e62cd7cfb13aaabbd2ad3a0abe9742be5c39a8ea8b68feb81f27a2187c6556a Nov 03 20:12:46.482486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.481133 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:46.482486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.481211 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:46.482486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.481241 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:46.732913 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.732881 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw"] Nov 03 20:12:46.734011 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:12:46.733986 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27621a94_aeca_461d_bf48_cddfb8c0ef53.slice/crio-8af2e3541232d886576f43c9eb06b82ac7942806cf07016136687b9b1c7ed513 WatchSource:0}: Error finding container 8af2e3541232d886576f43c9eb06b82ac7942806cf07016136687b9b1c7ed513: Status 404 returned error can't find the container with id 8af2e3541232d886576f43c9eb06b82ac7942806cf07016136687b9b1c7ed513 Nov 03 20:12:46.736519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.736491 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:46.736593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.736553 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:46.736593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.736586 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:46.796817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.796781 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" event={"ID":"27621a94-aeca-461d-bf48-cddfb8c0ef53","Type":"ContainerStarted","Data":"8af2e3541232d886576f43c9eb06b82ac7942806cf07016136687b9b1c7ed513"} Nov 03 20:12:46.816032 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.815446 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" event={"ID":"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0","Type":"ContainerStarted","Data":"1905035c712798dc2fbc6816f8719f2c4c1681b4e5260eb8e2bb08ca79a5d03a"} Nov 03 20:12:46.816032 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.815516 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" event={"ID":"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0","Type":"ContainerStarted","Data":"8e62cd7cfb13aaabbd2ad3a0abe9742be5c39a8ea8b68feb81f27a2187c6556a"} Nov 03 20:12:46.838450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:46.838362 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" podStartSLOduration=1.83834944 podStartE2EDuration="1.83834944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:46.837901755 +0000 UTC m=+945.213498502" watchObservedRunningTime="2026-11-03 20:12:46.83834944 +0000 UTC m=+945.213946187" Nov 03 20:12:47.189066 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:47.189029 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:47.189731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:47.189707 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:12:47.189884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:47.189853 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" podUID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 20:12:47.836934 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:47.836899 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" event={"ID":"27621a94-aeca-461d-bf48-cddfb8c0ef53","Type":"ContainerStarted","Data":"860c65d54d7a53b68c663a8edd9a28c488a93d082eba2cc085eca2a697d3a7a8"} Nov 03 20:12:47.865418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:47.863868 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" podStartSLOduration=1.863846482 podStartE2EDuration="1.863846482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:47.859954882 +0000 UTC m=+946.235551631" watchObservedRunningTime="2026-11-03 20:12:47.863846482 +0000 UTC m=+946.239443230" Nov 03 20:12:48.187553 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.185857 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:12:48.187553 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.185932 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" podUID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 20:12:48.433405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.433191 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:48.435024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.434909 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:12:48.435024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.434961 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" podUID="27621a94-aeca-461d-bf48-cddfb8c0ef53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 20:12:48.625985 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.625945 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr"] Nov 03 20:12:48.636202 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.636175 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" Nov 03 20:12:48.642239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.642210 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:12:48.642437 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.642323 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:12:48.645309 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.645286 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr"] Nov 03 20:12:48.737436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.736793 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-985bt\" (UniqueName: \"kubernetes.io/projected/a305e5f2-c91f-44fd-9568-150cbdda93da-kube-api-access-985bt\") pod \"maas-api-revoke-sub-2495bf59-m99fr\" (UID: \"a305e5f2-c91f-44fd-9568-150cbdda93da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" Nov 03 20:12:48.737436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.736848 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2495bf59-m99fr\" (UID: \"a305e5f2-c91f-44fd-9568-150cbdda93da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" Nov 03 20:12:48.737436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.737032 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2495bf59-m99fr\" (UID: \"a305e5f2-c91f-44fd-9568-150cbdda93da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" Nov 03 20:12:48.838076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.838033 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2495bf59-m99fr\" (UID: \"a305e5f2-c91f-44fd-9568-150cbdda93da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" Nov 03 20:12:48.838276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.838112 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-985bt\" (UniqueName: \"kubernetes.io/projected/a305e5f2-c91f-44fd-9568-150cbdda93da-kube-api-access-985bt\") pod \"maas-api-revoke-sub-2495bf59-m99fr\" (UID: \"a305e5f2-c91f-44fd-9568-150cbdda93da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" Nov 03 20:12:48.838276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.838164 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2495bf59-m99fr\" (UID: \"a305e5f2-c91f-44fd-9568-150cbdda93da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" Nov 03 20:12:48.840488 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.840459 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2495bf59-m99fr\" (UID: \"a305e5f2-c91f-44fd-9568-150cbdda93da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" Nov 03 20:12:48.847103 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.847074 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2495bf59-m99fr\" (UID: \"a305e5f2-c91f-44fd-9568-150cbdda93da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" Nov 03 20:12:48.852829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.852809 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-985bt\" (UniqueName: \"kubernetes.io/projected/a305e5f2-c91f-44fd-9568-150cbdda93da-kube-api-access-985bt\") pod \"maas-api-revoke-sub-2495bf59-m99fr\" (UID: \"a305e5f2-c91f-44fd-9568-150cbdda93da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" Nov 03 20:12:48.996459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:48.994568 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" Nov 03 20:12:49.187411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:49.184791 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:12:49.187411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:49.184853 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" podUID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 20:12:49.434421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:49.433139 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:12:49.434421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:49.433206 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" podUID="27621a94-aeca-461d-bf48-cddfb8c0ef53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 20:12:49.448661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:49.448633 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr"] Nov 03 20:12:49.919413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:49.901733 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" event={"ID":"a305e5f2-c91f-44fd-9568-150cbdda93da","Type":"ContainerStarted","Data":"31c625646e38e664940a1efbc977b12b23b405b355009803540d194aa8a5e097"} Nov 03 20:12:50.185411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:50.185350 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:12:50.185612 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:50.185431 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" podUID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 20:12:50.433882 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:50.433831 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:12:50.434121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:50.433908 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" podUID="27621a94-aeca-461d-bf48-cddfb8c0ef53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 20:12:50.919169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:50.919005 4457 generic.go:356] "Generic (PLEG): container finished" podID="a305e5f2-c91f-44fd-9568-150cbdda93da" containerID="6fa19c3e2c8af374d3b8f3b0b776ad20003b4a36e444da7a972eb868c56c0809" exitCode=0 Nov 03 20:12:50.919169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:50.919088 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" event={"ID":"a305e5f2-c91f-44fd-9568-150cbdda93da","Type":"ContainerDied","Data":"6fa19c3e2c8af374d3b8f3b0b776ad20003b4a36e444da7a972eb868c56c0809"} Nov 03 20:12:51.192411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:51.191776 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:12:51.192411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:51.191839 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" podUID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 20:12:51.433109 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:51.433079 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:12:51.433279 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:51.433133 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" podUID="27621a94-aeca-461d-bf48-cddfb8c0ef53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 20:12:52.244881 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.231959 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:12:52.244881 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.232111 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" podUID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 20:12:52.331155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.330836 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" Nov 03 20:12:52.442942 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.442905 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-xks-ca\") pod \"a305e5f2-c91f-44fd-9568-150cbdda93da\" (UID: \"a305e5f2-c91f-44fd-9568-150cbdda93da\") " Nov 03 20:12:52.443129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.442969 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a305e5f2-c91f-44fd-9568-150cbdda93da-kube-api-access-985bt\" (UniqueName: \"kubernetes.io/projected/a305e5f2-c91f-44fd-9568-150cbdda93da-kube-api-access-985bt\") pod \"a305e5f2-c91f-44fd-9568-150cbdda93da\" (UID: \"a305e5f2-c91f-44fd-9568-150cbdda93da\") " Nov 03 20:12:52.443129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.443007 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-openshift-ca\") pod \"a305e5f2-c91f-44fd-9568-150cbdda93da\" (UID: \"a305e5f2-c91f-44fd-9568-150cbdda93da\") " Nov 03 20:12:52.443284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.443256 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:12:52.443363 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.443322 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" podUID="27621a94-aeca-461d-bf48-cddfb8c0ef53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 20:12:52.444324 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.444299 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-openshift-ca" pod "a305e5f2-c91f-44fd-9568-150cbdda93da" (UID: "a305e5f2-c91f-44fd-9568-150cbdda93da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:12:52.446084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.446059 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-xks-ca" pod "a305e5f2-c91f-44fd-9568-150cbdda93da" (UID: "a305e5f2-c91f-44fd-9568-150cbdda93da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:12:52.446207 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.446082 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a305e5f2-c91f-44fd-9568-150cbdda93da-kube-api-access-985bt" pod "a305e5f2-c91f-44fd-9568-150cbdda93da" (UID: "a305e5f2-c91f-44fd-9568-150cbdda93da"). InnerVolumeSpecName "kube-api-access-985bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:12:52.545267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.545243 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:52.545447 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.545269 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-985bt\" (UniqueName: \"kubernetes.io/projected/a305e5f2-c91f-44fd-9568-150cbdda93da-kube-api-access-985bt\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:52.545447 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.545280 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a305e5f2-c91f-44fd-9568-150cbdda93da-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:12:52.937537 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.937438 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" Nov 03 20:12:52.937537 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.937511 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr" event={"ID":"a305e5f2-c91f-44fd-9568-150cbdda93da","Type":"ContainerDied","Data":"31c625646e38e664940a1efbc977b12b23b405b355009803540d194aa8a5e097"} Nov 03 20:12:52.937712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:52.937566 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31c625646e38e664940a1efbc977b12b23b405b355009803540d194aa8a5e097" Nov 03 20:12:53.184991 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:53.184955 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 03 20:12:53.185176 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:53.185009 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" podUID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 03 20:12:53.436765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:53.433099 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Nov 03 20:12:53.436765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:53.433162 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" podUID="27621a94-aeca-461d-bf48-cddfb8c0ef53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Nov 03 20:12:54.188783 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:54.188754 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:54.189053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:54.189023 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:54.189694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:54.189675 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:12:54.437461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:54.437430 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:54.437931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:54.437827 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:54.438776 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:54.438752 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:12:55.619363 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.619327 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb"] Nov 03 20:12:55.620071 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.620049 4457 cpu_manager.go:414] "containerMap: removing container" podUID="a305e5f2-c91f-44fd-9568-150cbdda93da" containerName="revoke-keys" Nov 03 20:12:55.620173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.620072 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a305e5f2-c91f-44fd-9568-150cbdda93da" containerName="revoke-keys" Nov 03 20:12:55.620235 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.620225 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="a305e5f2-c91f-44fd-9568-150cbdda93da" containerName="revoke-keys" Nov 03 20:12:55.623815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.623794 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.632212 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.632191 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-62221d-openshift-default-dockercfg-tqhkb\"" Nov 03 20:12:55.639712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.639688 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb"] Nov 03 20:12:55.795533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.795496 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.795533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.795542 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.795798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.795640 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.795798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.795667 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.795898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.795789 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.795898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.795867 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.795898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.795892 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.796047 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.795928 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.796047 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.795954 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzlzv\" (UniqueName: \"kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-kube-api-access-pzlzv\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.897303 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.897208 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.897303 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.897254 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.897587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.897329 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.897587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.897362 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.897587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.897511 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.897587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.897567 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.897795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.897603 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.897795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.897657 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.897795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.897687 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzlzv\" (UniqueName: \"kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-kube-api-access-pzlzv\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.898008 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.897984 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.898082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.898054 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.898139 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.898065 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.898297 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.898275 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.898667 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.898646 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.900341 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.900313 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.900663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.900640 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.909774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.909751 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.910050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.910030 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzlzv\" (UniqueName: \"kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-kube-api-access-pzlzv\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:55.943718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:55.943682 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:56.244064 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:56.244034 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb"] Nov 03 20:12:56.245355 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:12:56.245325 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92dfdb31_f6cb_4e35_a1c4_4dffb94f37ab.slice/crio-b08cd45e6c018a1a6b4967bb9f31198f795f4fe72cb96c4ea624fb29611ea6ee WatchSource:0}: Error finding container b08cd45e6c018a1a6b4967bb9f31198f795f4fe72cb96c4ea624fb29611ea6ee: Status 404 returned error can't find the container with id b08cd45e6c018a1a6b4967bb9f31198f795f4fe72cb96c4ea624fb29611ea6ee Nov 03 20:12:56.248225 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:56.248188 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:56.248315 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:56.248273 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:56.248403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:56.248315 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:12:56.986644 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:56.986593 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" event={"ID":"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab","Type":"ContainerStarted","Data":"da60cc90637ea437b3c8d97c02b9fdbb1e2727789243830d722351d1890d204f"} Nov 03 20:12:56.987183 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:56.986681 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" event={"ID":"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab","Type":"ContainerStarted","Data":"b08cd45e6c018a1a6b4967bb9f31198f795f4fe72cb96c4ea624fb29611ea6ee"} Nov 03 20:12:57.018033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.017971 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" podStartSLOduration=2.017951898 podStartE2EDuration="2.017951898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:57.015738557 +0000 UTC m=+955.391335305" watchObservedRunningTime="2026-11-03 20:12:57.017951898 +0000 UTC m=+955.393548646" Nov 03 20:12:57.308728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.308691 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g"] Nov 03 20:12:57.314081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.314049 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.316399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.315849 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-model-cache\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.316399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.315908 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-dshm\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.316399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.315959 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zqfs\" (UniqueName: \"kubernetes.io/projected/22be1486-517e-4055-8e00-c24c944eb960-kube-api-access-2zqfs\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.316399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.316033 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-tmp-dir\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.316399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.316071 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-home\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.316399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.316094 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22be1486-517e-4055-8e00-c24c944eb960-tls-certs\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.318023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.317966 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a6f98b\"/\"distinct-2-w5-a6f98b-kserve-self-signed-certs\"" Nov 03 20:12:57.321788 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.321761 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g"] Nov 03 20:12:57.420382 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.420277 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zqfs\" (UniqueName: \"kubernetes.io/projected/22be1486-517e-4055-8e00-c24c944eb960-kube-api-access-2zqfs\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.420952 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.420751 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-tmp-dir\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.420952 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.420824 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-home\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.420952 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.420848 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22be1486-517e-4055-8e00-c24c944eb960-tls-certs\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.421538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.421516 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-tmp-dir\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.422294 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.422272 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-home\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.422864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.422577 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-model-cache\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.423026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.423009 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-dshm\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.423424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.422808 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-model-cache\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.427557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.426682 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22be1486-517e-4055-8e00-c24c944eb960-tls-certs\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.427868 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.427831 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-dshm\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.432431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.432351 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zqfs\" (UniqueName: \"kubernetes.io/projected/22be1486-517e-4055-8e00-c24c944eb960-kube-api-access-2zqfs\") pod \"distinct-2-w5-a6f98b-kserve-78786759c9-cg99g\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.633246 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.633141 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:57.944698 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.944614 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:12:57.946211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.946184 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:12:57.946314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:57.946235 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" podUID="92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 20:12:58.021432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:58.021405 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g"] Nov 03 20:12:58.944347 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:58.944306 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:12:58.944558 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:58.944364 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" podUID="92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 20:12:59.005207 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:59.005160 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" event={"ID":"22be1486-517e-4055-8e00-c24c944eb960","Type":"ContainerStarted","Data":"870e7329d87ec9006d0522e43377aaeecb7559197086d2a7b046f0d62274a2af"} Nov 03 20:12:59.005421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:59.005220 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" event={"ID":"22be1486-517e-4055-8e00-c24c944eb960","Type":"ContainerStarted","Data":"5dfb17eaeb2aeb1a47482e8e5ac521d58a0a5a56db1a19ff527e3b535a935f22"} Nov 03 20:12:59.025123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:59.025073 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podStartSLOduration=2.025059151 podStartE2EDuration="2.025059151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:12:59.023612082 +0000 UTC m=+957.399208829" watchObservedRunningTime="2026-11-03 20:12:59.025059151 +0000 UTC m=+957.400655898" Nov 03 20:12:59.633761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:59.633723 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:12:59.944274 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:59.944198 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:12:59.944274 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:12:59.944258 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" podUID="92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 20:13:00.946982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:00.946944 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:13:00.947645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:00.947598 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" podUID="92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 20:13:01.949407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:01.947881 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:13:01.949407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:01.947943 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" podUID="92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 20:13:02.944639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:02.944608 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Nov 03 20:13:02.944832 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:02.944667 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" podUID="92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Nov 03 20:13:03.824548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:03.824517 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq"] Nov 03 20:13:03.830257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:03.830231 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq"] Nov 03 20:13:03.830470 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:03.830362 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:03.835680 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:03.835657 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-62221d\"/\"distinct-w3-62221d-kserve-self-signed-certs\"" Nov 03 20:13:03.957597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:03.957564 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tmp-dir\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:03.957803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:03.957607 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-model-cache\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:03.957803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:03.957634 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9wql\" (UniqueName: \"kubernetes.io/projected/5c089cbb-1f1d-4a87-97c1-b77de0012dda-kube-api-access-s9wql\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:03.957803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:03.957662 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-home\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:03.957803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:03.957681 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-dshm\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:03.957803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:03.957715 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tls-certs\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.066421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.060203 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tmp-dir\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.066421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.060266 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-model-cache\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.066421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.060303 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9wql\" (UniqueName: \"kubernetes.io/projected/5c089cbb-1f1d-4a87-97c1-b77de0012dda-kube-api-access-s9wql\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.066421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.060350 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-home\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.066421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.060379 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-dshm\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.066421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.060447 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tls-certs\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.066421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.061617 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tmp-dir\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.066421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.061858 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-model-cache\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.066421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.062098 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-home\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.066421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.064402 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-dshm\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.066421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.065181 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tls-certs\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.076272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.076186 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9wql\" (UniqueName: \"kubernetes.io/projected/5c089cbb-1f1d-4a87-97c1-b77de0012dda-kube-api-access-s9wql\") pod \"distinct-w3-62221d-kserve-5d5ccc7667-kr2cq\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.146593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.146553 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:04.495580 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.486821 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq"] Nov 03 20:13:04.497526 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:13:04.497490 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c089cbb_1f1d_4a87_97c1_b77de0012dda.slice/crio-9c2b07d66526d2f410579cb6873b1f3497e8f844f0d26a715221da472cdd49da WatchSource:0}: Error finding container 9c2b07d66526d2f410579cb6873b1f3497e8f844f0d26a715221da472cdd49da: Status 404 returned error can't find the container with id 9c2b07d66526d2f410579cb6873b1f3497e8f844f0d26a715221da472cdd49da Nov 03 20:13:04.509648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.509617 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:13:04.510632 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.510609 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:13:04.512628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:04.512604 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:13:05.224234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:05.224201 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" event={"ID":"5c089cbb-1f1d-4a87-97c1-b77de0012dda","Type":"ContainerStarted","Data":"807d14e1c3eb133760224cc4d11775d5a50ca0b66655acc0adafd7b740568f07"} Nov 03 20:13:05.224234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:05.224243 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" event={"ID":"5c089cbb-1f1d-4a87-97c1-b77de0012dda","Type":"ContainerStarted","Data":"9c2b07d66526d2f410579cb6873b1f3497e8f844f0d26a715221da472cdd49da"} Nov 03 20:13:05.246347 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:05.246294 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podStartSLOduration=2.246281347 podStartE2EDuration="2.246281347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:05.243238622 +0000 UTC m=+963.618835370" watchObservedRunningTime="2026-11-03 20:13:05.246281347 +0000 UTC m=+963.621878094" Nov 03 20:13:06.146516 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:13:06.146476 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:13:06.147608 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:06.147595 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:07.633588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:07.633543 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:13:07.646007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:07.645982 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:13:08.267085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:08.267053 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:13:10.242636 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.242600 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp"] Nov 03 20:13:10.248869 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.248839 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.255755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.255727 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d412cf\"/\"distinct-w6-d412cf-kserve-self-signed-certs\"" Nov 03 20:13:10.277505 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.277470 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp"] Nov 03 20:13:10.284032 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.284000 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-dshm\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.284285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.284264 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v28h5\" (UniqueName: \"kubernetes.io/projected/008491db-6f31-4ea1-9cbf-f9f779af78ea-kube-api-access-v28h5\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.284484 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.284465 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/008491db-6f31-4ea1-9cbf-f9f779af78ea-tls-certs\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.284715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.284697 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-home\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.284912 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.284894 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-model-cache\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.285070 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.285053 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-tmp-dir\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.387444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.386868 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-home\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.387444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.386965 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-model-cache\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.387444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.386993 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-tmp-dir\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.387444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.387095 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-dshm\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.387444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.387124 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v28h5\" (UniqueName: \"kubernetes.io/projected/008491db-6f31-4ea1-9cbf-f9f779af78ea-kube-api-access-v28h5\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.387444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.387171 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/008491db-6f31-4ea1-9cbf-f9f779af78ea-tls-certs\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.391821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.391788 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-home\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.392061 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.392041 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-model-cache\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.392298 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.392277 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-tmp-dir\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.398287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.398261 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/008491db-6f31-4ea1-9cbf-f9f779af78ea-tls-certs\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.402617 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.402591 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-dshm\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.425173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.423222 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v28h5\" (UniqueName: \"kubernetes.io/projected/008491db-6f31-4ea1-9cbf-f9f779af78ea-kube-api-access-v28h5\") pod \"distinct-w6-d412cf-kserve-6dff556f-mgbsp\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.578803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.578762 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:10.708316 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.707324 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25"] Nov 03 20:13:10.713756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.713716 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.714379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.714347 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25"] Nov 03 20:13:10.801198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.801151 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.801430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.801258 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.801430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.801350 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a580169-1ed3-4194-9bee-19a153b22984-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.801790 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.801758 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.801913 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.801811 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a580169-1ed3-4194-9bee-19a153b22984-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.801978 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.801918 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.801978 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.801953 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.802084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.802026 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btlcq\" (UniqueName: \"kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-kube-api-access-btlcq\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.802084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.802067 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.893268 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.893237 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp"] Nov 03 20:13:10.894456 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:13:10.894425 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod008491db_6f31_4ea1_9cbf_f9f779af78ea.slice/crio-b9da716f0beeec0533a452557595f5420b9c96f6946725689be676f4ce23a871 WatchSource:0}: Error finding container b9da716f0beeec0533a452557595f5420b9c96f6946725689be676f4ce23a871: Status 404 returned error can't find the container with id b9da716f0beeec0533a452557595f5420b9c96f6946725689be676f4ce23a871 Nov 03 20:13:10.903700 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.903673 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.903817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.903723 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a580169-1ed3-4194-9bee-19a153b22984-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.903817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.903787 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.903976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.903818 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.903976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.903858 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btlcq\" (UniqueName: \"kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-kube-api-access-btlcq\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.903976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.903889 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.904133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.903974 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.904133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.904025 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.904133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.904077 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a580169-1ed3-4194-9bee-19a153b22984-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.904981 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.904839 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a580169-1ed3-4194-9bee-19a153b22984-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.905232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.905207 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.906286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.906057 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.906593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.906505 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.906800 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.906772 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.910445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.910403 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.910539 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.910484 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a580169-1ed3-4194-9bee-19a153b22984-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.918096 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.918075 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:10.918423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:10.918371 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btlcq\" (UniqueName: \"kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-kube-api-access-btlcq\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:11.033594 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:11.033558 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:11.464815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:11.464717 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25"] Nov 03 20:13:11.478167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:11.478133 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" event={"ID":"008491db-6f31-4ea1-9cbf-f9f779af78ea","Type":"ContainerStarted","Data":"2bfec98d165f28e2a730aa0a33f5a026f669cb0b9cea36b537d87bf37a6208d3"} Nov 03 20:13:11.478647 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:11.478608 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" event={"ID":"008491db-6f31-4ea1-9cbf-f9f779af78ea","Type":"ContainerStarted","Data":"b9da716f0beeec0533a452557595f5420b9c96f6946725689be676f4ce23a871"} Nov 03 20:13:11.490467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:11.490429 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:11.491066 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:11.491038 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:11.491690 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:11.491666 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:11.500261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:11.500205 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podStartSLOduration=1.5001887329999999 podStartE2EDuration="1.500188733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:11.499424481 +0000 UTC m=+969.875021248" watchObservedRunningTime="2026-11-03 20:13:11.500188733 +0000 UTC m=+969.875785481" Nov 03 20:13:11.579755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:11.579686 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:12.499973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:12.499939 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" event={"ID":"3a580169-1ed3-4194-9bee-19a153b22984","Type":"ContainerStarted","Data":"1b452f46f109d18918c1ab574c8a41d34a67fbf058244d32eab1ad1efa12fd97"} Nov 03 20:13:12.500605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:12.500578 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" event={"ID":"3a580169-1ed3-4194-9bee-19a153b22984","Type":"ContainerStarted","Data":"ce3fded768478ac43138a40c1eb3c8b4513bcad205324e4eb9e08ccbc85a2e1d"} Nov 03 20:13:12.553151 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:12.553092 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" podStartSLOduration=2.553073507 podStartE2EDuration="2.553073507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:12.532595937 +0000 UTC m=+970.908192686" watchObservedRunningTime="2026-11-03 20:13:12.553073507 +0000 UTC m=+970.928670255" Nov 03 20:13:13.035305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:13.035264 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:13.038547 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:13.038514 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:13:13.038771 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:13.038739 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" podUID="3a580169-1ed3-4194-9bee-19a153b22984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 20:13:14.034246 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:14.034205 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:13:14.034747 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:14.034268 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" podUID="3a580169-1ed3-4194-9bee-19a153b22984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 20:13:14.146966 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:14.146928 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:14.194921 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:14.194886 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:14.536922 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:14.536892 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:15.034877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.034836 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:13:15.035373 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.034895 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" podUID="3a580169-1ed3-4194-9bee-19a153b22984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 20:13:15.700828 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.700788 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95"] Nov 03 20:13:15.706190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.706159 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.712281 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.712231 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95"] Nov 03 20:13:15.716714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.716365 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8013d9\"/\"distinct-2-w1-8013d9-kserve-self-signed-certs\"" Nov 03 20:13:15.785413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.785354 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkc7\" (UniqueName: \"kubernetes.io/projected/e6387668-070a-49e7-ab69-62ca763db030-kube-api-access-8jkc7\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.785590 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.785440 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-dshm\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.785590 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.785579 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-tmp-dir\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.785733 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.785636 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-model-cache\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.785733 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.785674 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6387668-070a-49e7-ab69-62ca763db030-tls-certs\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.785840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.785758 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-home\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.898181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.898128 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-home\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.898408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.898237 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkc7\" (UniqueName: \"kubernetes.io/projected/e6387668-070a-49e7-ab69-62ca763db030-kube-api-access-8jkc7\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.898408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.898287 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-dshm\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.898767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.898429 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-tmp-dir\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.898767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.898486 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-model-cache\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.898767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.898522 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6387668-070a-49e7-ab69-62ca763db030-tls-certs\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.907303 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.899850 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-tmp-dir\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.907303 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.900694 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-home\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.907303 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.900922 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-model-cache\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.911429 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.910694 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6387668-070a-49e7-ab69-62ca763db030-tls-certs\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.924914 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.924827 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn"] Nov 03 20:13:15.931025 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.930989 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkc7\" (UniqueName: \"kubernetes.io/projected/e6387668-070a-49e7-ab69-62ca763db030-kube-api-access-8jkc7\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.942434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.937948 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-dshm\") pod \"distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:15.963467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.957605 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq"] Nov 03 20:13:15.963648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:15.963580 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.006106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.005500 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.006106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.005548 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.006106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.005579 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.006106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.005619 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szk7m\" (UniqueName: \"kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-kube-api-access-szk7m\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.006106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.005655 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.006106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.005713 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.006106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.005763 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.006106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.005817 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.006106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.005917 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.015408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.010258 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn"] Nov 03 20:13:16.018550 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.012567 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.055743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.035991 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq"] Nov 03 20:13:16.093410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.048305 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:16.094162 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.048449 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:13:16.094162 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.093746 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" podUID="3a580169-1ed3-4194-9bee-19a153b22984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 20:13:16.132491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.130777 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.132491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.130853 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh"] Nov 03 20:13:16.132491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.131440 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.132491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.131749 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.132827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.130857 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.132827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.132783 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.132938 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.132828 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.132938 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.132876 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szk7m\" (UniqueName: \"kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-kube-api-access-szk7m\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.132938 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.132914 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.133094 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.132979 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.133094 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.133039 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.133196 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.133107 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.134872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.134834 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.135155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.135110 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.135637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.135617 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.137647 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.137300 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.137647 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.137368 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.137960 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.137908 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.144931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.144904 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh"] Nov 03 20:13:16.147318 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.147181 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szk7m\" (UniqueName: \"kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-kube-api-access-szk7m\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.147652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.147631 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.234123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.234092 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.234225 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.234141 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.234449 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.234265 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.234449 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.234296 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.234449 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.234420 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10eb980a-3ddd-4425-9636-7b44724c76ed-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.234641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.234510 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.234641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.234592 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbzgw\" (UniqueName: \"kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-kube-api-access-hbzgw\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.234751 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.234703 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.234812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.234752 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.336542 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.336510 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.336695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.336554 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.336695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.336604 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.336695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.336637 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbzgw\" (UniqueName: \"kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-kube-api-access-hbzgw\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.336695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.336679 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.336859 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.336703 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.336859 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.336722 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0fbd33-7b13-4702-b80d-a102f28187be-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.336859 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.336756 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.336859 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.336789 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.336859 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.336815 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.336859 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.336840 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.337078 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.336886 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn2fp\" (UniqueName: \"kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-kube-api-access-zn2fp\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.337078 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.336950 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.337280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.337188 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.338479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.338433 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.338754 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.338730 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.338818 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.338798 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.339007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.338987 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.339167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.339150 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.339312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.339292 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.339373 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.339328 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.339373 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.339364 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10eb980a-3ddd-4425-9636-7b44724c76ed-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.340150 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.340111 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10eb980a-3ddd-4425-9636-7b44724c76ed-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.342047 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.342025 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.343709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.343685 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.354837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.354810 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.365408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.355576 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbzgw\" (UniqueName: \"kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-kube-api-access-hbzgw\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.440812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.440769 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.440988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.440853 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.440988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.440881 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.441077 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.440985 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.441077 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.441040 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.441253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.441225 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0fbd33-7b13-4702-b80d-a102f28187be-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.441350 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.441279 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.441431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.441410 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.441485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.441422 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.441544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.441490 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.441544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.441519 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.441646 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.441541 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn2fp\" (UniqueName: \"kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-kube-api-access-zn2fp\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.441695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.441665 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.441975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.441953 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0fbd33-7b13-4702-b80d-a102f28187be-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.443790 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.443767 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.443852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.443841 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.445848 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.445828 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:16.454641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.454616 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.454641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.454632 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn2fp\" (UniqueName: \"kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-kube-api-access-zn2fp\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.519137 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.519064 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95"] Nov 03 20:13:16.520497 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:13:16.520473 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6387668_070a_49e7_ab69_62ca763db030.slice/crio-b3c7cb79990d4714292a567688d89276c8a75d8a157779881f4c3106e5d094d8 WatchSource:0}: Error finding container b3c7cb79990d4714292a567688d89276c8a75d8a157779881f4c3106e5d094d8: Status 404 returned error can't find the container with id b3c7cb79990d4714292a567688d89276c8a75d8a157779881f4c3106e5d094d8 Nov 03 20:13:16.533089 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.533063 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:16.542006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.541982 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:16.548323 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.548287 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" event={"ID":"e6387668-070a-49e7-ab69-62ca763db030","Type":"ContainerStarted","Data":"b3c7cb79990d4714292a567688d89276c8a75d8a157779881f4c3106e5d094d8"} Nov 03 20:13:16.832457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.832420 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn"] Nov 03 20:13:16.845897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.843308 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:16.846201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.846168 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:16.846358 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:16.846343 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:17.034755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.034650 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:13:17.034755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.034704 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" podUID="3a580169-1ed3-4194-9bee-19a153b22984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 20:13:17.042116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.041335 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq"] Nov 03 20:13:17.046106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.046073 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:17.046211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.046160 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:17.046585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.046207 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:17.048791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.048621 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh"] Nov 03 20:13:17.048791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.048633 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:17.048791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.048689 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:17.048791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.048717 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:17.602822 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.602779 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" event={"ID":"e6387668-070a-49e7-ab69-62ca763db030","Type":"ContainerStarted","Data":"07cd3f211bbbce3c253aba8d4fe8d2644239cc11dbd553da41a9dc3cb91d740c"} Nov 03 20:13:17.609027 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.608985 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" event={"ID":"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41","Type":"ContainerStarted","Data":"a66d83f00f3997b54a401d05dbd7d363d7900571f45991e670b1c91afb45b807"} Nov 03 20:13:17.609200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.609050 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" event={"ID":"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41","Type":"ContainerStarted","Data":"db58586a287ca574d120e537e8a7d9a50a13f19c26e254277359b1ca3af616ed"} Nov 03 20:13:17.614635 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.614601 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" event={"ID":"ab0fbd33-7b13-4702-b80d-a102f28187be","Type":"ContainerStarted","Data":"e16da9771721620bbd31c0ecda2c81095f716e4ff819726fd8f57416105118db"} Nov 03 20:13:17.618189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.618163 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" event={"ID":"ab0fbd33-7b13-4702-b80d-a102f28187be","Type":"ContainerStarted","Data":"d9dfd87305379ac114d625de4fb8cb03cfc0b976e4d9cce6d27045e331e40000"} Nov 03 20:13:17.618420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.618402 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" event={"ID":"10eb980a-3ddd-4425-9636-7b44724c76ed","Type":"ContainerStarted","Data":"002132b6698812c012b45b3990d7f780ee7659ea5e86d3b7bcbe5c3d7fab9987"} Nov 03 20:13:17.618572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.618557 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" event={"ID":"10eb980a-3ddd-4425-9636-7b44724c76ed","Type":"ContainerStarted","Data":"3e98af5b167a9904f8fc88339d2cabc3dabcd525473b160195941da49901ac0a"} Nov 03 20:13:17.630296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.630243 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podStartSLOduration=2.630226395 podStartE2EDuration="2.630226395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:17.623465195 +0000 UTC m=+975.999061943" watchObservedRunningTime="2026-11-03 20:13:17.630226395 +0000 UTC m=+976.005823146" Nov 03 20:13:17.649430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.648644 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" podStartSLOduration=2.6486313949999998 podStartE2EDuration="2.648631395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:17.64802188 +0000 UTC m=+976.023618630" watchObservedRunningTime="2026-11-03 20:13:17.648631395 +0000 UTC m=+976.024228142" Nov 03 20:13:17.720300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.720234 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" podStartSLOduration=2.720215578 podStartE2EDuration="2.720215578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:17.691705209 +0000 UTC m=+976.067301958" watchObservedRunningTime="2026-11-03 20:13:17.720215578 +0000 UTC m=+976.095812327" Nov 03 20:13:17.726816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:17.726747 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" podStartSLOduration=2.726732884 podStartE2EDuration="2.726732884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:17.718875882 +0000 UTC m=+976.094472630" watchObservedRunningTime="2026-11-03 20:13:17.726732884 +0000 UTC m=+976.102329632" Nov 03 20:13:18.036205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:18.036113 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:13:18.036368 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:18.036191 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" podUID="3a580169-1ed3-4194-9bee-19a153b22984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 20:13:18.054682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:18.051952 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:18.453411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:18.453313 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:18.453669 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:18.453545 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:13:18.453669 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:18.453591 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" podUID="2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 20:13:18.535410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:18.535255 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:18.535622 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:18.535457 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:18.535622 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:18.535504 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 20:13:18.556412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:18.547754 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:18.556412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:18.547882 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:18.556412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:18.547925 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:13:19.040410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:19.035854 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 03 20:13:19.040410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:19.035914 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" podUID="3a580169-1ed3-4194-9bee-19a153b22984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 03 20:13:19.451430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:19.450798 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:13:19.451430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:19.450865 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" podUID="2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 20:13:19.525411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:19.524577 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:19.538820 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:19.537917 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:19.538820 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:19.538075 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 20:13:19.543611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:19.542728 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:19.543611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:19.542803 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:13:19.564413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:19.556956 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:20.353414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:20.352654 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:20.373696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:20.373647 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:20.452411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:20.451174 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:13:20.452411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:20.451240 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" podUID="2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 20:13:20.537412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:20.536369 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:20.537412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:20.536455 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 20:13:20.546405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:20.546277 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:20.546405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:20.546333 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:13:20.583007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:20.582083 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:20.606613 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:20.606478 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:20.608509 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:20.608490 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:20.617806 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:20.617703 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:13:20.860031 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:20.859440 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 03 20:13:21.396414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.366899 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:13:21.396414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.367455 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:13:21.406458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.402449 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:21.474436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.449345 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:13:21.474436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.449422 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" podUID="2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 20:13:21.475268 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.474844 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:21.503279 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.502859 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:13:21.537440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.537367 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:21.537646 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.537441 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 20:13:21.538161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.537793 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:21.565867 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.565812 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:21.599428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.568962 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:21.599781 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.599685 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:13:21.623527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:21.623482 4457 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:22.063414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:22.062844 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:13:22.297418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:22.293911 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:22.452703 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:22.450846 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:13:22.452703 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:22.450905 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" podUID="2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 20:13:22.541998 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:22.541932 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:22.542301 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:22.542268 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 20:13:22.544800 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:22.544770 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:22.545002 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:22.544980 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:13:22.582555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:22.582506 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:22.594439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:22.583255 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:23.202411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.178979 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:23.284195 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.284148 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:23.295739 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.295695 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:13:23.330897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.323428 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:13:23.529473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.477542 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:13:23.529473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.477609 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" podUID="2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 20:13:23.551731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.532461 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:23.551731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.551142 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:23.551731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.551204 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:13:23.551731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.551290 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:23.551731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.551312 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 20:13:23.599419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.574235 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:23.599419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.576954 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:23.664410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.648476 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:23.768188 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:23.756063 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:13:24.296540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.296489 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:13:24.354812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.352106 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:13:24.454784 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.454747 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body= Nov 03 20:13:24.455062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.455032 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" podUID="2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" Nov 03 20:13:24.540328 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.540204 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:24.540328 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.540265 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 20:13:24.547191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.547010 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:24.547191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.547074 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:13:24.604430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.576990 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:13:24.604430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.577403 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:13:24.678523 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.678470 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:24.698443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.694089 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:13:24.775548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.775513 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:13:24.809480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:24.783939 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:13:25.524866 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:25.524229 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:25.526156 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:25.525091 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:25.527628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:25.527608 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:13:25.534749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:25.534719 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 03 20:13:25.534872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:25.534784 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 03 20:13:25.546869 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:25.546837 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 03 20:13:25.547529 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:25.547480 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 03 20:13:26.048770 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:26.048712 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:26.070761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:26.070729 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:26.219041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:26.218705 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:26.626812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:26.626774 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:26.628685 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:26.628666 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:26.631132 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:26.631115 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:13:26.648224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:26.648193 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:26.649857 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:26.649833 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:26.651066 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:26.651047 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:13:30.592999 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:30.592952 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:30.605064 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:30.605025 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:31.006796 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.006696 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8"] Nov 03 20:13:31.011847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.011816 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.030158 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.030120 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8"] Nov 03 20:13:31.062284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.062251 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2690f13c-4a95-48d3-9b47-598b87d5d992-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.062472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.062295 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.062562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.062479 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.062623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.062562 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.062623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.062609 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8tj8\" (UniqueName: \"kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-kube-api-access-r8tj8\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.062732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.062676 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.062791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.062733 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.062844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.062793 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.062895 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.062861 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.117619 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.117580 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg"] Nov 03 20:13:31.121785 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.121763 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.135431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.135379 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg"] Nov 03 20:13:31.164661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.164620 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.164865 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.164670 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.164865 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.164706 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8tj8\" (UniqueName: \"kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-kube-api-access-r8tj8\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.164865 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.164770 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.164865 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.164795 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27abf71d-0887-4f90-a511-54719c9194db-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.164865 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.164822 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.164865 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.164863 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.165197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.164914 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.165197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.164937 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27abf71d-0887-4f90-a511-54719c9194db-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.165197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.164975 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.165197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.165021 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8l5k\" (UniqueName: \"kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-kube-api-access-k8l5k\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.165197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.165091 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2690f13c-4a95-48d3-9b47-598b87d5d992-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.165197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.165119 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.165197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.165147 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.165883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.165206 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.165883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.165296 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.165883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.165327 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.165883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.165349 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.165883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.165372 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.165883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.165381 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.165883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.165792 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.166248 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.166076 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.166248 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.166102 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2690f13c-4a95-48d3-9b47-598b87d5d992-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.169430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.168507 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.169430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.168920 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.194481 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.186779 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.194481 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.193962 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8tj8\" (UniqueName: \"kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-kube-api-access-r8tj8\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.199396 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.199351 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j"] Nov 03 20:13:31.204958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.204935 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.212995 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.212915 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j"] Nov 03 20:13:31.266892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.266791 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.266892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.266850 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.267123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.266906 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.267123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.266946 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.267123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.266993 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.267123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.267019 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6l2d\" (UniqueName: \"kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-kube-api-access-r6l2d\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.267123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.267077 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.267530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.267125 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27abf71d-0887-4f90-a511-54719c9194db-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.267530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.267202 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.267530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.267226 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27abf71d-0887-4f90-a511-54719c9194db-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.267530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.267299 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8l5k\" (UniqueName: \"kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-kube-api-access-k8l5k\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.267530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.267324 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.267530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.267457 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.267530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.267489 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.267881 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.267557 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.267881 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.267599 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.267881 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.267624 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.267881 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.267656 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.268284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.268261 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.268569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.268549 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.274447 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.270005 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.274447 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.270023 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.274447 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.270514 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27abf71d-0887-4f90-a511-54719c9194db-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.278227 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.278197 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27abf71d-0887-4f90-a511-54719c9194db-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.279109 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.278907 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.285427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.281667 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8l5k\" (UniqueName: \"kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-kube-api-access-k8l5k\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.285427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.284778 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.334891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.334850 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:31.368586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.368552 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.368741 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.368693 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.368801 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.368755 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.368855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.368804 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.368855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.368841 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.368961 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.368871 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.368961 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.368924 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.368961 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.368955 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.369111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.369046 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6l2d\" (UniqueName: \"kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-kube-api-access-r6l2d\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.369797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.369771 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.372941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.372296 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.372941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.372589 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.372941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.372657 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.373172 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.373103 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.373520 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.373477 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.380779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.380675 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.383845 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.383673 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6l2d\" (UniqueName: \"kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-kube-api-access-r6l2d\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.385139 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.385119 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.446951 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.446664 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:31.527083 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.526989 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:31.652155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.652093 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8"] Nov 03 20:13:31.654249 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:13:31.654219 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2690f13c_4a95_48d3_9b47_598b87d5d992.slice/crio-34621d62042da4e2c0d9bc1aabf76ffc067a7b8bc63bf86bc507ed5a91b96942 WatchSource:0}: Error finding container 34621d62042da4e2c0d9bc1aabf76ffc067a7b8bc63bf86bc507ed5a91b96942: Status 404 returned error can't find the container with id 34621d62042da4e2c0d9bc1aabf76ffc067a7b8bc63bf86bc507ed5a91b96942 Nov 03 20:13:31.657555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.656895 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.657555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.656955 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.657555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.656995 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.776236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.776039 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg"] Nov 03 20:13:31.776623 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:13:31.776569 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27abf71d_0887_4f90_a511_54719c9194db.slice/crio-74d434c652c3712f5e9e01758b76b89872f3783caf6625123da3d6c9ceab3f34 WatchSource:0}: Error finding container 74d434c652c3712f5e9e01758b76b89872f3783caf6625123da3d6c9ceab3f34: Status 404 returned error can't find the container with id 74d434c652c3712f5e9e01758b76b89872f3783caf6625123da3d6c9ceab3f34 Nov 03 20:13:31.779486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.779448 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.779629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.779614 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.779721 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.779710 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.914256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.913590 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.914256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.913681 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.914256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.913735 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:13:31.917745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:31.917137 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j"] Nov 03 20:13:32.249970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.239119 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" event={"ID":"27abf71d-0887-4f90-a511-54719c9194db","Type":"ContainerStarted","Data":"5aa6f4e731589257eb2a9efe391c3d2891d83ef37a4a05c68d25049f89219b8b"} Nov 03 20:13:32.249970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.239174 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" event={"ID":"27abf71d-0887-4f90-a511-54719c9194db","Type":"ContainerStarted","Data":"74d434c652c3712f5e9e01758b76b89872f3783caf6625123da3d6c9ceab3f34"} Nov 03 20:13:32.249970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.242159 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" event={"ID":"0af92d22-ef86-41f9-a91e-2f5b6653bad6","Type":"ContainerStarted","Data":"290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38"} Nov 03 20:13:32.249970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.242205 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" event={"ID":"0af92d22-ef86-41f9-a91e-2f5b6653bad6","Type":"ContainerStarted","Data":"68f9c7749b164a15c33051f2fab5def386dbf112c2eaa21808fa57f375999fe5"} Nov 03 20:13:32.251472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.250725 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" event={"ID":"2690f13c-4a95-48d3-9b47-598b87d5d992","Type":"ContainerStarted","Data":"273d86fa3f9797ed1a1b5109dc913dbb73f8b93b8bebdffe6938fe0e2236ca4e"} Nov 03 20:13:32.251472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.250773 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" event={"ID":"2690f13c-4a95-48d3-9b47-598b87d5d992","Type":"ContainerStarted","Data":"34621d62042da4e2c0d9bc1aabf76ffc067a7b8bc63bf86bc507ed5a91b96942"} Nov 03 20:13:32.270866 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.270798 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podStartSLOduration=1.270779907 podStartE2EDuration="1.270779907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:32.267620461 +0000 UTC m=+990.643217208" watchObservedRunningTime="2026-11-03 20:13:32.270779907 +0000 UTC m=+990.646376655" Nov 03 20:13:32.306595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.306534 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" podStartSLOduration=1.306515385 podStartE2EDuration="1.306515385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:32.29289543 +0000 UTC m=+990.668492193" watchObservedRunningTime="2026-11-03 20:13:32.306515385 +0000 UTC m=+990.682112133" Nov 03 20:13:32.323758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.323698 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podStartSLOduration=2.323657067 podStartE2EDuration="2.323657067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:32.319840428 +0000 UTC m=+990.695437190" watchObservedRunningTime="2026-11-03 20:13:32.323657067 +0000 UTC m=+990.699253819" Nov 03 20:13:32.335223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.335186 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:32.336762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.336736 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:32.336875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.336779 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:13:32.448420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.448352 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:32.449880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.449845 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:13:32.450004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.449910 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:13:32.528112 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:32.528031 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:33.337111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:33.336794 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:33.337111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:33.336858 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:13:33.448825 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:33.448785 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:13:33.449119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:33.449084 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:13:33.529583 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:33.529545 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:13:33.529869 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:33.529840 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:13:34.342410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:34.335742 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:34.342410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:34.335802 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:13:34.397413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:34.396250 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:13:34.448411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:34.447914 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:13:34.448411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:34.447978 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:13:34.530412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:34.529720 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:13:34.530412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:34.529779 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:13:34.795472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:34.780525 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.031412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.030752 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.082469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.072801 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.201616 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.181154 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.231454 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.230481 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.250628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.249546 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.250628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.249862 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:13:35.335862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.335767 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:35.336122 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.336089 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:13:35.357144 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.357098 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.376299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.375472 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.397408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.396470 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:13:35.397408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.396922 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:35.399599 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.397727 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:13:35.448420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.448367 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:13:35.448686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.448654 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:13:35.531803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.531771 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:13:35.532907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.532875 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:13:35.556236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.556187 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:13:35.577409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.576453 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:13:35.950131 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.934487 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:36.002410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:35.994150 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:13:36.068413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.055256 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:13:36.073727 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.073691 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:13:36.075059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.075035 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:13:36.221493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.208757 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:13:36.284419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.261104 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:13:36.416739 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.383046 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:36.421205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.389059 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:36.421955 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.398373 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:13:36.422144 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.421496 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:13:36.422917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.422896 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:36.423150 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.423135 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:13:36.423339 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.423313 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:13:36.573963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.559494 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:36.574501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.561359 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:13:36.574694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.574669 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:13:36.574811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.561632 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:13:36.574926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.574904 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:13:36.584410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.575358 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:13:36.605420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.603488 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:36.978483 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.954153 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:36.978483 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:36.954579 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:13:37.000352 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:13:37.000258 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:13:37.128438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.051474 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": context deadline exceeded" Nov 03 20:13:37.139856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.139809 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:13:37.242515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.236667 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.242515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.237027 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:37.275344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.275280 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.350791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.343764 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:37.350791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.343824 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:13:37.405415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.361634 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.426530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.426484 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.491443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.380584 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:37.491822 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.421466 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:13:37.544189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.541553 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:13:37.544189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.541619 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:13:37.561431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.544560 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:13:37.561431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.544612 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:13:37.575569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.575528 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:37.727125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.727070 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:13:38.035947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.728211 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Nov 03 20:13:38.036229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:37.747618 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:13:38.167956 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.166275 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:13:38.179941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.179885 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:13:38.251414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.245585 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:38.259595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.259549 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:38.296506 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.296448 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:38.311114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.307531 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:38.346900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.345624 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:38.346900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.345701 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:13:38.350544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.349695 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:38.350544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.349801 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:13:38.350544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.349869 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:13:38.350544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.349944 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:13:38.350544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.350002 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:13:38.741800 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.741684 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:38.791508 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.791456 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:38.816410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.792315 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:13:38.934787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.934758 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:39.010469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.816703 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:13:39.010469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.827270 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:13:39.010469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.831517 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:13:39.010469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.993773 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:13:39.010469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.816646 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:39.010469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:38.905686 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:39.010469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.000237 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:39.010469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.000646 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:13:39.010469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.009197 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:13:39.010469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.009270 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:13:39.228413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.226752 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:39.228413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.227170 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:39.228413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.227501 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:13:39.296239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.296199 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:39.297207 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.297176 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:39.298065 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.298036 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:39.377350 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.336819 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:39.401425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.400818 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:13:39.447701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.447569 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:13:39.447701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.447630 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:13:39.553480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.537246 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:13:39.553480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.537307 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:13:39.625064 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.625027 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:13:39.718755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.718725 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:13:39.802055 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.801115 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:39.803815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.803737 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:39.994520 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.994478 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:40.005448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.997506 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:13:40.038448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.997545 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:13:40.038448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:39.997585 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:13:40.038448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.000456 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:40.038448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.001579 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:40.138131 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.138028 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:40.143749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.143718 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:13:40.233083 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.231241 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:13:40.326418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.300135 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:40.339418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.327478 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:13:40.383559 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.383511 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:40.443469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.442438 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:40.443469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.442499 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:13:40.448906 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.448798 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:13:40.448906 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.448853 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:13:40.792475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.750561 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:13:40.807537 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.807487 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:13:40.810445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.810403 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:40.832464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.812180 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:40.834049 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.826510 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:40.834203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:40.833040 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:41.011335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.011278 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:41.021045 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.020995 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:13:41.021600 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.021566 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:41.022273 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.022246 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:41.236123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.233588 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:41.341434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.335843 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:13:41.341434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.337158 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:41.341434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.337204 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:13:41.341434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.338631 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:41.449410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.445491 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:41.473422 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.458663 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:13:41.473737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.473696 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:13:41.513661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.513575 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:13:41.534546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.534505 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:13:41.534831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.534800 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:13:41.598543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.598494 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:41.734572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.734527 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:13:41.862451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.846226 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:41.905143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:41.903495 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:42.027527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.027479 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:42.028115 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.028085 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:13:42.028531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.028508 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:13:42.028884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.028862 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:13:42.144841 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.144638 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:42.160605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.159037 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:13:42.169461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.168901 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:13:42.177023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.177000 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:13:42.265527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.265483 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:13:42.297317 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.297287 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:13:42.299284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.299264 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:13:42.336592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.336235 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:42.398753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.393250 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 03 20:13:42.398753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.393308 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 03 20:13:42.447617 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.447573 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 03 20:13:42.448176 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.448120 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 03 20:13:42.506652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.506623 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:13:42.510631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.510607 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:13:42.517485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.517465 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:13:42.553068 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.544535 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:13:42.553068 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.539725 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 03 20:13:42.553068 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:42.545414 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 03 20:13:43.919790 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:43.919750 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:43.920949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:43.920925 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:43.921973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:43.921950 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:13:44.235671 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:44.235583 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:44.236939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:44.236919 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:44.242761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:44.242730 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:13:44.402094 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:44.402044 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:13:44.402378 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:44.402334 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:44.636006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:44.635954 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:44.636517 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:44.636486 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:44.637762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:44.637739 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:13:53.903409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:53.892277 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc"] Nov 03 20:13:53.903409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:53.897720 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:53.907494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:53.906774 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-b1b190\"/\"distinct-2-w0-b1b190-kserve-self-signed-certs\"" Nov 03 20:13:53.913975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:53.911975 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc"] Nov 03 20:13:54.022409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.022157 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-home\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.022409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.022198 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-dshm\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.022409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.022243 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tmp-dir\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.022409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.022331 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-model-cache\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.022409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.022364 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz8bl\" (UniqueName: \"kubernetes.io/projected/a97a4814-275f-4b05-88a0-b2ab3cd6305c-kube-api-access-gz8bl\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.043287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.040771 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tls-certs\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.043287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.042620 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:13:54.311497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.295472 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:54.311497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.306996 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-home\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.311497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.307051 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-dshm\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.311497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.307096 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tmp-dir\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.311497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.307217 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-model-cache\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.311497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.307253 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz8bl\" (UniqueName: \"kubernetes.io/projected/a97a4814-275f-4b05-88a0-b2ab3cd6305c-kube-api-access-gz8bl\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.311497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.307376 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tls-certs\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.315411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.312877 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tmp-dir\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.315411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.313162 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-home\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.315411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.313768 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-model-cache\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.414538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.399057 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf"] Nov 03 20:13:54.445411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.437755 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:54.498411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.446263 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-dshm\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.498411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.483933 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tls-certs\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.600479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.580202 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz8bl\" (UniqueName: \"kubernetes.io/projected/a97a4814-275f-4b05-88a0-b2ab3cd6305c-kube-api-access-gz8bl\") pod \"distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.666410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.665994 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" Nov 03 20:13:54.691412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.684286 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:13:54.691412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.684465 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:13:54.808166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.808128 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b223ef8a-rqvwf\" (UID: \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" Nov 03 20:13:54.812778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.812742 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97kn2\" (UniqueName: \"kubernetes.io/projected/ab5c653a-d229-4ff8-b937-fdba3b5c7012-kube-api-access-97kn2\") pod \"maas-api-revoke-sub-b223ef8a-rqvwf\" (UID: \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" Nov 03 20:13:54.813171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.813148 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b223ef8a-rqvwf\" (UID: \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" Nov 03 20:13:54.854728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.846616 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf"] Nov 03 20:13:54.857172 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.857142 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:54.953029 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.952977 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b223ef8a-rqvwf\" (UID: \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" Nov 03 20:13:54.953745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.953714 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97kn2\" (UniqueName: \"kubernetes.io/projected/ab5c653a-d229-4ff8-b937-fdba3b5c7012-kube-api-access-97kn2\") pod \"maas-api-revoke-sub-b223ef8a-rqvwf\" (UID: \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" Nov 03 20:13:54.953958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.953939 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b223ef8a-rqvwf\" (UID: \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" Nov 03 20:13:54.955366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.955306 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b223ef8a-rqvwf\" (UID: \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" Nov 03 20:13:55.001411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:54.986114 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b223ef8a-rqvwf\" (UID: \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" Nov 03 20:13:55.005410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:55.003111 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97kn2\" (UniqueName: \"kubernetes.io/projected/ab5c653a-d229-4ff8-b937-fdba3b5c7012-kube-api-access-97kn2\") pod \"maas-api-revoke-sub-b223ef8a-rqvwf\" (UID: \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" Nov 03 20:13:55.043409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:55.042500 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.043409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:55.042902 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Nov 03 20:13:55.050414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:55.043735 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.050414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:55.044070 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:13:55.251464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:55.251110 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" Nov 03 20:13:55.319652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:55.297108 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.319652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:55.298040 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:13:55.322033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:55.320433 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.325408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:55.324702 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:55.546786 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:55.546745 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:13:55.734792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:55.734548 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:56.594782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:56.594748 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:13:56.595450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:56.595428 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc"] Nov 03 20:13:56.603167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:56.603132 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:13:56.706616 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:56.706590 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:13:57.145624 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:13:57.145586 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5c653a_d229_4ff8_b937_fdba3b5c7012.slice/crio-0e3df29e24a5a454cdfa70deb2004ac0b99d0942de2f33a31a0267c2a7f6c41c WatchSource:0}: Error finding container 0e3df29e24a5a454cdfa70deb2004ac0b99d0942de2f33a31a0267c2a7f6c41c: Status 404 returned error can't find the container with id 0e3df29e24a5a454cdfa70deb2004ac0b99d0942de2f33a31a0267c2a7f6c41c Nov 03 20:13:57.146340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:57.146312 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf"] Nov 03 20:13:57.369946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:57.369905 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" event={"ID":"ab5c653a-d229-4ff8-b937-fdba3b5c7012","Type":"ContainerStarted","Data":"0e3df29e24a5a454cdfa70deb2004ac0b99d0942de2f33a31a0267c2a7f6c41c"} Nov 03 20:13:57.371876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:57.371837 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" event={"ID":"a97a4814-275f-4b05-88a0-b2ab3cd6305c","Type":"ContainerStarted","Data":"953b275330478d98d952bc48b9ea397bfb3a09a69ba174a8e6ac627786266ab5"} Nov 03 20:13:57.371876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:57.371873 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" event={"ID":"a97a4814-275f-4b05-88a0-b2ab3cd6305c","Type":"ContainerStarted","Data":"c6056f0291a4ac55028e128b4352d65264cf428ab80228f68f0cf7ab124233dd"} Nov 03 20:13:57.397623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:57.397508 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podStartSLOduration=4.397495246 podStartE2EDuration="4.397495246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:13:57.394469368 +0000 UTC m=+1015.770066116" watchObservedRunningTime="2026-11-03 20:13:57.397495246 +0000 UTC m=+1015.773091992" Nov 03 20:13:57.862189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:57.862158 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:13:58.383469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:58.383424 4457 generic.go:356] "Generic (PLEG): container finished" podID="ab5c653a-d229-4ff8-b937-fdba3b5c7012" containerID="caa54df55478235b92fb31225db644f995a102d60225ad4424e6422f01a337b7" exitCode=0 Nov 03 20:13:58.383641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:58.383491 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" event={"ID":"ab5c653a-d229-4ff8-b937-fdba3b5c7012","Type":"ContainerDied","Data":"caa54df55478235b92fb31225db644f995a102d60225ad4424e6422f01a337b7"} Nov 03 20:13:59.603404 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:59.598108 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" Nov 03 20:13:59.699448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:59.699332 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-xks-ca\") pod \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\" (UID: \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\") " Nov 03 20:13:59.699599 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:59.699489 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5c653a-d229-4ff8-b937-fdba3b5c7012-kube-api-access-97kn2\" (UniqueName: \"kubernetes.io/projected/ab5c653a-d229-4ff8-b937-fdba3b5c7012-kube-api-access-97kn2\") pod \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\" (UID: \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\") " Nov 03 20:13:59.699650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:59.699618 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-openshift-ca\") pod \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\" (UID: \"ab5c653a-d229-4ff8-b937-fdba3b5c7012\") " Nov 03 20:13:59.700018 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:59.699985 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-openshift-ca" pod "ab5c653a-d229-4ff8-b937-fdba3b5c7012" (UID: "ab5c653a-d229-4ff8-b937-fdba3b5c7012"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:13:59.700274 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:59.700255 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:59.702191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:59.702157 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5c653a-d229-4ff8-b937-fdba3b5c7012-kube-api-access-97kn2" pod "ab5c653a-d229-4ff8-b937-fdba3b5c7012" (UID: "ab5c653a-d229-4ff8-b937-fdba3b5c7012"). InnerVolumeSpecName "kube-api-access-97kn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:13:59.702277 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:59.702249 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-xks-ca" pod "ab5c653a-d229-4ff8-b937-fdba3b5c7012" (UID: "ab5c653a-d229-4ff8-b937-fdba3b5c7012"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:13:59.802261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:59.802228 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab5c653a-d229-4ff8-b937-fdba3b5c7012-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:13:59.802261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:13:59.802263 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97kn2\" (UniqueName: \"kubernetes.io/projected/ab5c653a-d229-4ff8-b937-fdba3b5c7012-kube-api-access-97kn2\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:00.405057 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:00.405030 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" Nov 03 20:14:00.405340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:00.405055 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf" event={"ID":"ab5c653a-d229-4ff8-b937-fdba3b5c7012","Type":"ContainerDied","Data":"0e3df29e24a5a454cdfa70deb2004ac0b99d0942de2f33a31a0267c2a7f6c41c"} Nov 03 20:14:00.405485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:00.405373 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e3df29e24a5a454cdfa70deb2004ac0b99d0942de2f33a31a0267c2a7f6c41c" Nov 03 20:14:01.044877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.044840 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg"] Nov 03 20:14:01.045471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.045448 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ab5c653a-d229-4ff8-b937-fdba3b5c7012" containerName="revoke-keys" Nov 03 20:14:01.045471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.045470 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5c653a-d229-4ff8-b937-fdba3b5c7012" containerName="revoke-keys" Nov 03 20:14:01.045799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.045779 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5c653a-d229-4ff8-b937-fdba3b5c7012" containerName="revoke-keys" Nov 03 20:14:01.049656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.049640 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.064633 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.064606 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg"] Nov 03 20:14:01.118212 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.118172 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdkb4\" (UniqueName: \"kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-kube-api-access-rdkb4\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.118417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.118324 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.118417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.118355 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.118540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.118497 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8658c4-9961-4acd-8180-20128324c476-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.118595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.118538 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.118595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.118565 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8658c4-9961-4acd-8180-20128324c476-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.118694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.118617 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.118694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.118679 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.118791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.118710 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.219652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.219614 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8658c4-9961-4acd-8180-20128324c476-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.219652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.219649 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.219911 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.219665 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8658c4-9961-4acd-8180-20128324c476-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.219911 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.219801 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.219911 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.219884 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.220058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.219928 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.220058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.220001 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdkb4\" (UniqueName: \"kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-kube-api-access-rdkb4\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.220186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.220169 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.220241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.220222 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.220291 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.220262 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-certs\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.220404 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.220366 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-data\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.220601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.220579 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8658c4-9961-4acd-8180-20128324c476-istiod-ca-cert\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.220694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.220636 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.220759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.220717 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-credential-socket\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.222141 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.222115 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-envoy\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.222459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.222437 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8658c4-9961-4acd-8180-20128324c476-istio-podinfo\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.232096 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.232066 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-istio-token\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.232252 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.232230 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdkb4\" (UniqueName: \"kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-kube-api-access-rdkb4\") pod \"e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.368814 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.368779 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:01.670537 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.670510 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg"] Nov 03 20:14:01.673378 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:14:01.673347 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8658c4_9961_4acd_8180_20128324c476.slice/crio-e08ef4544bb40aa4c8cb63efc811457bfae1ad6022bfddc206137aa115898aa1 WatchSource:0}: Error finding container e08ef4544bb40aa4c8cb63efc811457bfae1ad6022bfddc206137aa115898aa1: Status 404 returned error can't find the container with id e08ef4544bb40aa4c8cb63efc811457bfae1ad6022bfddc206137aa115898aa1 Nov 03 20:14:01.677412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.677364 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:01.677507 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.677445 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:01.677507 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:01.677478 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:02.425254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:02.425200 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" event={"ID":"5d8658c4-9961-4acd-8180-20128324c476","Type":"ContainerStarted","Data":"dacb3def3c367b61616568cc0dd63b85354b96c33d7249e3414d1cdd8f9ed201"} Nov 03 20:14:02.425777 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:02.425278 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" event={"ID":"5d8658c4-9961-4acd-8180-20128324c476","Type":"ContainerStarted","Data":"e08ef4544bb40aa4c8cb63efc811457bfae1ad6022bfddc206137aa115898aa1"} Nov 03 20:14:02.474014 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:02.473077 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" podStartSLOduration=1.473059383 podStartE2EDuration="1.473059383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:02.450737558 +0000 UTC m=+1020.826334305" watchObservedRunningTime="2026-11-03 20:14:02.473059383 +0000 UTC m=+1020.848656130" Nov 03 20:14:03.370948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:03.370918 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:03.375516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:03.375485 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:14:03.376237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:03.376206 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" podUID="5d8658c4-9961-4acd-8180-20128324c476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 20:14:04.371834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:04.371792 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:14:04.373527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:04.373494 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" podUID="5d8658c4-9961-4acd-8180-20128324c476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 20:14:04.871424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:04.870771 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:14:04.898155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:04.898121 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:14:05.373843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:05.373799 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:14:05.374497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:05.374459 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" podUID="5d8658c4-9961-4acd-8180-20128324c476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 20:14:05.491310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:05.491279 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:14:06.214090 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:14:06.214046 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:14:06.370847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:06.370809 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:14:06.371158 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:06.371107 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" podUID="5d8658c4-9961-4acd-8180-20128324c476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 20:14:07.372801 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:07.372756 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:14:07.373431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:07.373375 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" podUID="5d8658c4-9961-4acd-8180-20128324c476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 20:14:08.369431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:08.369375 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 03 20:14:08.369610 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:08.369474 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" podUID="5d8658c4-9961-4acd-8180-20128324c476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 03 20:14:09.374576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:09.374544 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:09.375058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:09.375043 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:09.376001 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:09.375975 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:14:14.356030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.355991 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l"] Nov 03 20:14:14.362788 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.362759 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.372909 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.372871 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l"] Nov 03 20:14:14.374150 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.374124 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-90ef9c\"/\"distinct-2-w2-90ef9c-kserve-self-signed-certs\"" Nov 03 20:14:14.454844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.454792 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdlxp\" (UniqueName: \"kubernetes.io/projected/289c7342-c7d8-4281-bcb2-164607ad83ea-kube-api-access-vdlxp\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.455023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.454860 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-dshm\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.455163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.455137 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-tmp-dir\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.455284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.455264 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-home\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.455334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.455308 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-model-cache\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.455378 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.455336 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289c7342-c7d8-4281-bcb2-164607ad83ea-tls-certs\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.557307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.557266 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-tmp-dir\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.557543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.557379 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-home\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.557543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.557447 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-model-cache\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.557543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.557474 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289c7342-c7d8-4281-bcb2-164607ad83ea-tls-certs\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.557730 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.557556 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdlxp\" (UniqueName: \"kubernetes.io/projected/289c7342-c7d8-4281-bcb2-164607ad83ea-kube-api-access-vdlxp\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.557730 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.557598 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-dshm\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.558795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.558764 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-tmp-dir\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.558984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.558815 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-home\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.558984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.558839 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-model-cache\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.560851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.560828 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-dshm\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.564733 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.564710 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289c7342-c7d8-4281-bcb2-164607ad83ea-tls-certs\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.571174 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.571106 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdlxp\" (UniqueName: \"kubernetes.io/projected/289c7342-c7d8-4281-bcb2-164607ad83ea-kube-api-access-vdlxp\") pod \"distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:14.691465 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:14.690559 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:15.069123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:15.069090 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l"] Nov 03 20:14:15.069899 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:14:15.069862 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod289c7342_c7d8_4281_bcb2_164607ad83ea.slice/crio-9f5d10bb8a9a87cb1a5052629bc650029ebf08941961f1e49a98c3e2300f32e6 WatchSource:0}: Error finding container 9f5d10bb8a9a87cb1a5052629bc650029ebf08941961f1e49a98c3e2300f32e6: Status 404 returned error can't find the container with id 9f5d10bb8a9a87cb1a5052629bc650029ebf08941961f1e49a98c3e2300f32e6 Nov 03 20:14:15.575040 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:15.575000 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" event={"ID":"289c7342-c7d8-4281-bcb2-164607ad83ea","Type":"ContainerStarted","Data":"be32a44234f74347c436d43d8250ac9fa87eb7764638404d550f9df4a4a8b6d9"} Nov 03 20:14:15.575546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:15.575051 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" event={"ID":"289c7342-c7d8-4281-bcb2-164607ad83ea","Type":"ContainerStarted","Data":"9f5d10bb8a9a87cb1a5052629bc650029ebf08941961f1e49a98c3e2300f32e6"} Nov 03 20:14:15.599140 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:15.599076 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podStartSLOduration=1.599057049 podStartE2EDuration="1.599057049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:15.593745394 +0000 UTC m=+1033.969342142" watchObservedRunningTime="2026-11-03 20:14:15.599057049 +0000 UTC m=+1033.974653797" Nov 03 20:14:15.692420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:15.692347 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:16.046063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.046019 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj"] Nov 03 20:14:16.054576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.054547 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.077921 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.077872 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj"] Nov 03 20:14:16.093692 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.093651 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.093840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.093700 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.093840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.093814 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.093925 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.093849 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj57r\" (UniqueName: \"kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-kube-api-access-cj57r\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.093925 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.093913 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.094038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.094023 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086323f3-b415-4716-94d4-ea06f7ada7ee-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.094119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.094103 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.094207 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.094193 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.094326 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.094312 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.195127 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.195094 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.195127 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.195130 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.195381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.195172 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.195381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.195206 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj57r\" (UniqueName: \"kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-kube-api-access-cj57r\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.195381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.195228 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.195381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.195265 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086323f3-b415-4716-94d4-ea06f7ada7ee-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.195381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.195339 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.195719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.195427 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.195719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.195478 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.195719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.195610 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-certs\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.195719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.195676 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.195919 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.195804 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-data\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.196152 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.196125 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086323f3-b415-4716-94d4-ea06f7ada7ee-istiod-ca-cert\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.196288 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.196258 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-credential-socket\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.197865 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.197844 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-podinfo\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.198118 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.198096 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-envoy\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.208605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.208568 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj57r\" (UniqueName: \"kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-kube-api-access-cj57r\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.208713 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.208623 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-token\") pod \"e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.389535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.389451 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:16.689976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.689943 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj"] Nov 03 20:14:16.692042 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.692002 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:16.692138 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.692092 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:16.692179 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:16.692136 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:17.680412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:17.676363 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" event={"ID":"086323f3-b415-4716-94d4-ea06f7ada7ee","Type":"ContainerStarted","Data":"639bc6eae804a3afbb34e0e83beed35a0196da389ac24057ab8f12b321164f75"} Nov 03 20:14:17.711412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:17.701520 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" event={"ID":"086323f3-b415-4716-94d4-ea06f7ada7ee","Type":"ContainerStarted","Data":"135af36d2106f628e70d8cd039967cca3036131ac408a7e8f787feb589a64380"} Nov 03 20:14:17.741129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:17.741053 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podStartSLOduration=1.7410310070000001 podStartE2EDuration="1.741031007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:17.726626086 +0000 UTC m=+1036.102222835" watchObservedRunningTime="2026-11-03 20:14:17.741031007 +0000 UTC m=+1036.116627755" Nov 03 20:14:18.397550 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:18.396445 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:18.397550 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:18.396494 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:14:18.397550 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:18.396541 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:14:19.253550 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:19.253497 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:19.391875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:19.391815 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:14:19.392083 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:19.391905 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:14:19.450411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:19.449546 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:19.464411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:19.455463 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:19.596546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:19.596489 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:19.621715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:19.621662 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:14:19.773550 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:19.773491 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:14:20.125288 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.125235 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.360448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.348650 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.408468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.401900 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:14:20.443499 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.410456 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:14:20.443499 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.418511 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:20.443499 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.419684 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:14:20.443499 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.422166 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:14:20.443499 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.422215 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:14:20.488412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.450761 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.488412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.451165 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:14:20.488412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.473463 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.488412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.473884 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:14:20.534003 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.494178 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.555451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.555405 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.612289 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.602784 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.618422 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.615355 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:14:20.776121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.776022 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.876427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.793057 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:14:20.876427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.805072 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:20.942413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:20.941488 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.059664 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.050484 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.158452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.117181 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:14:21.330803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.330596 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:14:21.392913 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.392879 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:14:21.396508 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.396471 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:14:21.402811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.402768 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:14:21.404628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.404605 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:14:21.413933 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.413889 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.426609 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.426563 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:14:21.509412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.435357 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:21.560849 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.458180 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:14:21.584595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.494483 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.585742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.494544 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.610421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.598124 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:14:21.629414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.593535 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.629414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.611891 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:14:21.686559 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.686506 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:21.988404 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:21.988288 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:14:22.043619 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:22.043576 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:14:22.276608 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:22.275869 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:14:22.402108 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:22.381108 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.458579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:22.458518 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.471726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:22.471672 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:22.631474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:22.534665 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.032479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:22.586582 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 03 20:14:23.032479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:22.629677 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:14:23.032479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:22.804777 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:14:23.032479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.030426 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.054431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.040586 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:14:23.148415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.059402 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.177415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.162913 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.299409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.298025 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.333412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.332704 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.358414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.349640 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:14:23.492434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.474541 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.492434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.478969 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.492434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.480971 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:14:23.579187 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:14:23.552502 4457 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.796s" Nov 03 20:14:23.579187 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.552970 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:14:23.579187 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.553338 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:14:23.579187 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.553685 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:14:23.579187 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.554016 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:14:23.579187 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.566429 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:23.603443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.594803 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:14:23.603443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.594863 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:14:23.603443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.595453 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:14:23.603443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.595488 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:14:23.732418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:23.719067 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:14:24.320623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.320589 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:14:24.320947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.320929 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:14:24.321088 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.321074 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:14:24.369478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.365803 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:14:24.401811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.401760 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:14:24.416072 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.402108 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:14:24.449996 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.449945 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:14:24.453049 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.453027 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:14:24.568372 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.563993 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:24.568372 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.564120 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:14:24.568372 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.566614 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:14:24.579678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.579052 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:24.579678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.579330 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:24.579678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.579594 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:14:24.585948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.585877 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:14:24.587309 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.587279 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:14:24.720592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.720558 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:24.751380 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.751352 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:14:24.758258 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.758230 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:14:24.806074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.791263 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:14:24.806074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.796405 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:14:24.806074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.800074 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:24.806074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.800442 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:14:24.806074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:24.804139 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:14:25.394879 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.394832 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:14:25.395080 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.394905 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:14:25.704940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.704855 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:25.838965 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.838927 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs"] Nov 03 20:14:25.844172 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.844148 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:25.911737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.911700 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs"] Nov 03 20:14:25.941734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.941688 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:25.942006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.941757 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:25.942006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.941895 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa9c354-722f-46a0-be30-c4683f7e2254-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:25.942147 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.942094 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:25.942147 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.942136 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:25.942238 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.942199 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:25.942779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.942280 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsp82\" (UniqueName: \"kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-kube-api-access-jsp82\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:25.942779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.942457 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:25.942779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:25.942521 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.030420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.030298 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp"] Nov 03 20:14:26.034934 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.034907 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" Nov 03 20:14:26.039494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.039465 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:14:26.040409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.039559 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:14:26.043338 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.043310 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.043482 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.043355 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.043482 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.043436 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.043620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.043489 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsp82\" (UniqueName: \"kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-kube-api-access-jsp82\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.043620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.043513 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.043620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.043534 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.043620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.043590 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.043620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.043617 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.044016 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.043681 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85q98\" (UniqueName: \"kubernetes.io/projected/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-kube-api-access-85q98\") pod \"maas-api-revoke-sub-b3293116-xjpbp\" (UID: \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" Nov 03 20:14:26.044016 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.043750 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa9c354-722f-46a0-be30-c4683f7e2254-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.044016 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.043816 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3293116-xjpbp\" (UID: \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" Nov 03 20:14:26.044016 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.043844 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3293116-xjpbp\" (UID: \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" Nov 03 20:14:26.044748 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.044721 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.044858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.044741 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-certs\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.045155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.045019 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-data\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.045441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.045414 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa9c354-722f-46a0-be30-c4683f7e2254-istiod-ca-cert\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.045693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.045670 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-credential-socket\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.047649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.047624 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp"] Nov 03 20:14:26.047757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.047708 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-envoy\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.048291 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.048274 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-podinfo\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.112163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.112033 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-token\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.113361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.113332 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsp82\" (UniqueName: \"kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-kube-api-access-jsp82\") pod \"e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.144781 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.144743 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85q98\" (UniqueName: \"kubernetes.io/projected/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-kube-api-access-85q98\") pod \"maas-api-revoke-sub-b3293116-xjpbp\" (UID: \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" Nov 03 20:14:26.144975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.144851 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3293116-xjpbp\" (UID: \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" Nov 03 20:14:26.144975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.144880 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3293116-xjpbp\" (UID: \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" Nov 03 20:14:26.145679 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.145654 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3293116-xjpbp\" (UID: \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" Nov 03 20:14:26.147228 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.147202 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3293116-xjpbp\" (UID: \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" Nov 03 20:14:26.193517 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.193483 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85q98\" (UniqueName: \"kubernetes.io/projected/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-kube-api-access-85q98\") pod \"maas-api-revoke-sub-b3293116-xjpbp\" (UID: \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" Nov 03 20:14:26.196287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.196258 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:26.354260 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.354229 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" Nov 03 20:14:26.401189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.400458 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:14:26.401189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.400516 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:14:26.613458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.608581 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs"] Nov 03 20:14:26.624026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.623977 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:26.624128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.624083 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:26.624184 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.624129 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:26.726170 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.726134 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" event={"ID":"7fa9c354-722f-46a0-be30-c4683f7e2254","Type":"ContainerStarted","Data":"1828f03b8b0164cee1e50db68efb2599b568b33d52a34d2701c3d3a7c97b27b8"} Nov 03 20:14:26.745138 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:26.745110 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp"] Nov 03 20:14:26.747965 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:14:26.747941 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a45915a_881d_40e3_bc6f_aa9c2f03ad06.slice/crio-292700a3f042a6299e73236fe7868e31f1b16318fbde383a6c572df11105bef4 WatchSource:0}: Error finding container 292700a3f042a6299e73236fe7868e31f1b16318fbde383a6c572df11105bef4: Status 404 returned error can't find the container with id 292700a3f042a6299e73236fe7868e31f1b16318fbde383a6c572df11105bef4 Nov 03 20:14:27.394145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:27.391191 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body= Nov 03 20:14:27.394145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:27.391289 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" Nov 03 20:14:27.736522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:27.736439 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" event={"ID":"6a45915a-881d-40e3-bc6f-aa9c2f03ad06","Type":"ContainerStarted","Data":"fac73a045bed2b38fae07233bbe55b24319cda3bc82505c28e02b93adb252dfe"} Nov 03 20:14:27.737029 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:27.737007 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" event={"ID":"6a45915a-881d-40e3-bc6f-aa9c2f03ad06","Type":"ContainerStarted","Data":"292700a3f042a6299e73236fe7868e31f1b16318fbde383a6c572df11105bef4"} Nov 03 20:14:27.738449 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:27.738421 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" event={"ID":"7fa9c354-722f-46a0-be30-c4683f7e2254","Type":"ContainerStarted","Data":"83594e7b8b89929813990979f540dfc2732b338a59fabac9756e4bd7f62b979b"} Nov 03 20:14:27.780676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:27.756828 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" podStartSLOduration=1.7568121639999998 podStartE2EDuration="1.756812164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:27.753777784 +0000 UTC m=+1046.129374529" watchObservedRunningTime="2026-11-03 20:14:27.756812164 +0000 UTC m=+1046.132408913" Nov 03 20:14:27.818041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:27.809787 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" podStartSLOduration=2.8097655550000002 podStartE2EDuration="2.809765555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:27.79301496 +0000 UTC m=+1046.168611708" watchObservedRunningTime="2026-11-03 20:14:27.809765555 +0000 UTC m=+1046.185362303" Nov 03 20:14:28.199412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:28.198681 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:28.201501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:28.200131 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:28.201501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:28.200206 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" podUID="7fa9c354-722f-46a0-be30-c4683f7e2254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:14:28.534817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:28.534722 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:28.535641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:28.535623 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:28.536637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:28.536614 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:14:28.748619 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:28.748581 4457 generic.go:356] "Generic (PLEG): container finished" podID="6a45915a-881d-40e3-bc6f-aa9c2f03ad06" containerID="fac73a045bed2b38fae07233bbe55b24319cda3bc82505c28e02b93adb252dfe" exitCode=0 Nov 03 20:14:28.749087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:28.748657 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" event={"ID":"6a45915a-881d-40e3-bc6f-aa9c2f03ad06","Type":"ContainerDied","Data":"fac73a045bed2b38fae07233bbe55b24319cda3bc82505c28e02b93adb252dfe"} Nov 03 20:14:29.205035 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:29.204697 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:29.205035 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:29.204779 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" podUID="7fa9c354-722f-46a0-be30-c4683f7e2254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:14:30.100270 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.100241 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" Nov 03 20:14:30.193428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.188707 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-xks-ca\") pod \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\" (UID: \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\") " Nov 03 20:14:30.193761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.193737 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-openshift-ca\") pod \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\" (UID: \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\") " Nov 03 20:14:30.194257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.194235 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-kube-api-access-85q98\" (UniqueName: \"kubernetes.io/projected/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-kube-api-access-85q98\") pod \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\" (UID: \"6a45915a-881d-40e3-bc6f-aa9c2f03ad06\") " Nov 03 20:14:30.195357 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.193582 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-xks-ca" pod "6a45915a-881d-40e3-bc6f-aa9c2f03ad06" (UID: "6a45915a-881d-40e3-bc6f-aa9c2f03ad06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:14:30.195357 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.194175 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-openshift-ca" pod "6a45915a-881d-40e3-bc6f-aa9c2f03ad06" (UID: "6a45915a-881d-40e3-bc6f-aa9c2f03ad06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:14:30.196803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.196775 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:30.196920 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.196832 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" podUID="7fa9c354-722f-46a0-be30-c4683f7e2254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:14:30.200812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.200788 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-kube-api-access-85q98" pod "6a45915a-881d-40e3-bc6f-aa9c2f03ad06" (UID: "6a45915a-881d-40e3-bc6f-aa9c2f03ad06"). InnerVolumeSpecName "kube-api-access-85q98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:14:30.296451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.296416 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:30.296642 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.296454 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:30.296642 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.296470 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85q98\" (UniqueName: \"kubernetes.io/projected/6a45915a-881d-40e3-bc6f-aa9c2f03ad06-kube-api-access-85q98\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:14:30.809316 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.807778 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" event={"ID":"6a45915a-881d-40e3-bc6f-aa9c2f03ad06","Type":"ContainerDied","Data":"292700a3f042a6299e73236fe7868e31f1b16318fbde383a6c572df11105bef4"} Nov 03 20:14:30.809316 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.807826 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="292700a3f042a6299e73236fe7868e31f1b16318fbde383a6c572df11105bef4" Nov 03 20:14:30.809316 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:30.807877 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp" Nov 03 20:14:31.187466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.179256 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s"] Nov 03 20:14:31.198074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.198034 4457 cpu_manager.go:414] "containerMap: removing container" podUID="6a45915a-881d-40e3-bc6f-aa9c2f03ad06" containerName="revoke-keys" Nov 03 20:14:31.198286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.198263 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a45915a-881d-40e3-bc6f-aa9c2f03ad06" containerName="revoke-keys" Nov 03 20:14:31.199071 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.199031 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a45915a-881d-40e3-bc6f-aa9c2f03ad06" containerName="revoke-keys" Nov 03 20:14:31.212667 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.211889 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:31.212667 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.211985 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" podUID="7fa9c354-722f-46a0-be30-c4683f7e2254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:14:31.219513 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.219484 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.223631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.223607 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf"] Nov 03 20:14:31.228433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.228413 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.230475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.230363 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s"] Nov 03 20:14:31.238670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.238642 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf"] Nov 03 20:14:31.319408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.318887 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh"] Nov 03 20:14:31.327286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.326901 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5721044-7f44-4a81-a212-abc21f39bc1e-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.327286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.326971 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r96h9\" (UniqueName: \"kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-kube-api-access-r96h9\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.327286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327011 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.327286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327059 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.327286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327091 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.327286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327153 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.327286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327193 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3062e080-8433-4f4b-b0e3-ffc0da402d66-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.327286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327242 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.327856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327310 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.327856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327346 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.327856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327375 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.327856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327442 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.327856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327491 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.327856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327530 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.327856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327555 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.327856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327581 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.327856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327622 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.327856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.327656 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bjwt\" (UniqueName: \"kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-kube-api-access-9bjwt\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.341108 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.341078 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh"] Nov 03 20:14:31.352490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.352456 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.360279 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.360248 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7"] Nov 03 20:14:31.396472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.388768 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7"] Nov 03 20:14:31.396472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.388897 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.428736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.428700 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.428901 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.428753 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.428901 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.428800 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.428901 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.428819 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.429075 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.428934 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.429075 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.428966 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.429075 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.428986 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.429075 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429020 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.429075 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429044 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.429337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429070 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.429337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429105 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c925c7-63db-46a9-a936-a0a56f28e396-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.429337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429132 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.429337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429170 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.429337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429187 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.429337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429207 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.429337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429252 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.429337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429275 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.429337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429296 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bjwt\" (UniqueName: \"kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-kube-api-access-9bjwt\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.429337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429330 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24n8w\" (UniqueName: \"kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-kube-api-access-24n8w\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.429916 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429358 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.429916 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429645 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpw4h\" (UniqueName: \"kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-kube-api-access-jpw4h\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.429916 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.429746 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-data\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.430079 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430008 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5721044-7f44-4a81-a212-abc21f39bc1e-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430192 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430214 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-certs\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430294 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96h9\" (UniqueName: \"kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-kube-api-access-r96h9\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430340 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430378 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430454 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430482 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14a467c-de34-4c09-837d-ef5fb99986b7-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430493 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430518 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430707 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c925c7-63db-46a9-a936-a0a56f28e396-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430821 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430852 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430901 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430931 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.430975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.430964 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.431876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.431001 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3062e080-8433-4f4b-b0e3-ffc0da402d66-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.433483 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.432357 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-credential-socket\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.433483 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.432989 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3062e080-8433-4f4b-b0e3-ffc0da402d66-istiod-ca-cert\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.433483 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.433194 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-podinfo\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.433483 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.433201 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5721044-7f44-4a81-a212-abc21f39bc1e-istiod-ca-cert\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.433483 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.433440 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-credential-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.434194 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.434152 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-data\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.434602 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.434562 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-socket\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.435000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.434978 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-envoy\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.435931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.435663 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-podinfo\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.435931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.435887 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-envoy\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.436707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.436686 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-certs\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.445832 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.445754 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r96h9\" (UniqueName: \"kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-kube-api-access-r96h9\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.446916 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.446867 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bjwt\" (UniqueName: \"kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-kube-api-access-9bjwt\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.448143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.448091 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-token\") pod \"e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.449222 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.449167 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-token\") pod \"e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.532707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.532674 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.532872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.532730 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14a467c-de34-4c09-837d-ef5fb99986b7-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.532872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.532756 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c925c7-63db-46a9-a936-a0a56f28e396-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.532872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.532782 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.532872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.532811 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.532872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.532850 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.533161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.532880 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.533161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.532938 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.533161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.532984 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.533161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533011 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.533161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533075 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.533161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533107 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.533161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533110 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-data\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.533161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533145 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c925c7-63db-46a9-a936-a0a56f28e396-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.535244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533226 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.535244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533256 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24n8w\" (UniqueName: \"kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-kube-api-access-24n8w\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.535244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533295 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.535244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533351 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpw4h\" (UniqueName: \"kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-kube-api-access-jpw4h\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.535244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533416 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.535244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533708 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-credential-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.535244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533763 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.535244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533808 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.535244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.533958 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.535244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.534012 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-socket\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.535244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.534158 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14a467c-de34-4c09-837d-ef5fb99986b7-istiod-ca-cert\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.535244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.534403 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.535244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.534801 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c925c7-63db-46a9-a936-a0a56f28e396-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.536298 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.536079 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:31.536715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.536445 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c925c7-63db-46a9-a936-a0a56f28e396-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.536715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.536674 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-certs\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.536902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.536880 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-podinfo\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.537745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.537718 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.538518 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.538495 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-envoy\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.546957 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.546638 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpw4h\" (UniqueName: \"kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-kube-api-access-jpw4h\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.547820 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.547636 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:31.548603 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.548577 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.549178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.549136 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-token\") pod \"e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.552164 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.551912 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24n8w\" (UniqueName: \"kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-kube-api-access-24n8w\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.721004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.720930 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:31.724817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.724786 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:31.932983 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.932942 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:31.935440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.935405 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:31.935555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.935473 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:31.938537 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.938509 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s"] Nov 03 20:14:31.997334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:31.997297 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf"] Nov 03 20:14:32.025814 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:14:32.025780 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5721044_7f44_4a81_a212_abc21f39bc1e.slice/crio-f1d64b4ba0a0b25663b0a1d9969feb3032b66a762c84af828e85220d770750d2 WatchSource:0}: Error finding container f1d64b4ba0a0b25663b0a1d9969feb3032b66a762c84af828e85220d770750d2: Status 404 returned error can't find the container with id f1d64b4ba0a0b25663b0a1d9969feb3032b66a762c84af828e85220d770750d2 Nov 03 20:14:32.033087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.033035 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.033579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.033550 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.033751 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.033731 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.197311 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.197274 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:32.197754 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.197330 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" podUID="7fa9c354-722f-46a0-be30-c4683f7e2254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:14:32.310600 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.310472 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7"] Nov 03 20:14:32.321412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.316698 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh"] Nov 03 20:14:32.334154 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:14:32.334118 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14a467c_de34_4c09_837d_ef5fb99986b7.slice/crio-00507a996046ff2f2ac4d30b96db6810ef4b7e4f1d2fab2bb997cf06d044511d WatchSource:0}: Error finding container 00507a996046ff2f2ac4d30b96db6810ef4b7e4f1d2fab2bb997cf06d044511d: Status 404 returned error can't find the container with id 00507a996046ff2f2ac4d30b96db6810ef4b7e4f1d2fab2bb997cf06d044511d Nov 03 20:14:32.334724 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.334683 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.334827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.334787 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.334886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.334845 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.343672 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.343640 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.344789 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.343857 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.344998 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.344963 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:14:32.851408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.850671 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" event={"ID":"3062e080-8433-4f4b-b0e3-ffc0da402d66","Type":"ContainerStarted","Data":"d636455cacf019a44c5dfccedba6a25e38f79387f31811a7af23cba6f12c6360"} Nov 03 20:14:32.851408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.850744 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" event={"ID":"3062e080-8433-4f4b-b0e3-ffc0da402d66","Type":"ContainerStarted","Data":"363e62a2709004a80e985df14f14236f27890e9561aacded12f65f57905910eb"} Nov 03 20:14:32.859404 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.854219 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" event={"ID":"62c925c7-63db-46a9-a936-a0a56f28e396","Type":"ContainerStarted","Data":"8be577314845fdde68c2420c09a95b2868a2f30cfe04215a565f30523bc146ad"} Nov 03 20:14:32.859404 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.854279 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" event={"ID":"62c925c7-63db-46a9-a936-a0a56f28e396","Type":"ContainerStarted","Data":"529bf57a082b5d21e1fa12d89dc794bafe7fd8b54727e8ccb8e8a94755972105"} Nov 03 20:14:32.865404 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.862549 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" event={"ID":"f14a467c-de34-4c09-837d-ef5fb99986b7","Type":"ContainerStarted","Data":"086b31bb286d6ace17871f003864f076fa7e04033a60988e6d5bb2fd8a240ab8"} Nov 03 20:14:32.865404 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.862605 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" event={"ID":"f14a467c-de34-4c09-837d-ef5fb99986b7","Type":"ContainerStarted","Data":"00507a996046ff2f2ac4d30b96db6810ef4b7e4f1d2fab2bb997cf06d044511d"} Nov 03 20:14:32.870410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.866486 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" event={"ID":"b5721044-7f44-4a81-a212-abc21f39bc1e","Type":"ContainerStarted","Data":"6478cfa8470743528dc74fde5bcc0d8ec021a4a6e5f596473acbabb2d1aaecdb"} Nov 03 20:14:32.870410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.866535 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" event={"ID":"b5721044-7f44-4a81-a212-abc21f39bc1e","Type":"ContainerStarted","Data":"f1d64b4ba0a0b25663b0a1d9969feb3032b66a762c84af828e85220d770750d2"} Nov 03 20:14:32.944412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.943847 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" podStartSLOduration=1.943824694 podStartE2EDuration="1.943824694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:32.900574475 +0000 UTC m=+1051.276171222" watchObservedRunningTime="2026-11-03 20:14:32.943824694 +0000 UTC m=+1051.319421443" Nov 03 20:14:32.975186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:32.975123 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podStartSLOduration=1.975104021 podStartE2EDuration="1.975104021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:32.972069282 +0000 UTC m=+1051.347666030" watchObservedRunningTime="2026-11-03 20:14:32.975104021 +0000 UTC m=+1051.350700770" Nov 03 20:14:33.024284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.024200 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" podStartSLOduration=2.024177692 podStartE2EDuration="2.024177692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:32.945172728 +0000 UTC m=+1051.320769489" watchObservedRunningTime="2026-11-03 20:14:33.024177692 +0000 UTC m=+1051.399774442" Nov 03 20:14:33.053033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.052974 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" podStartSLOduration=2.052953637 podStartE2EDuration="2.052953637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:14:32.997243929 +0000 UTC m=+1051.372840679" watchObservedRunningTime="2026-11-03 20:14:33.052953637 +0000 UTC m=+1051.428550385" Nov 03 20:14:33.201615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.201025 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:33.201615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.201095 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" podUID="7fa9c354-722f-46a0-be30-c4683f7e2254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:14:33.545062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.544660 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:33.564412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.546852 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:33.564412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.546915 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 20:14:33.564412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.552574 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:33.565907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.565433 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:33.565907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.565495 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:14:33.732421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.732204 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:33.732421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.732341 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:33.732421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.732402 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 20:14:33.765767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.734027 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:33.765767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.734172 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:14:33.765767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:33.734209 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 20:14:34.206414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.203999 4457 patch_prober.go:29] interesting pod/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.206414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.204058 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" podUID="7fa9c354-722f-46a0-be30-c4683f7e2254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 03 20:14:34.304535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.304483 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:34.490471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.489490 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:34.537423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.536657 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:34.541417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.538023 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.541417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.538071 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 20:14:34.573411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.571764 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:34.573411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.572706 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.573411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.572757 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:14:34.599412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.599021 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:34.599590 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.599451 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:34.712413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.711483 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:34.737411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.736489 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:34.741474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.741033 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.741474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.741088 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 20:14:34.741474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.741366 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:34.741474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.741461 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 20:14:34.778305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.777507 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:34.811668 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.811001 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:14:34.898764 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.897600 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:14:34.994486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:34.994359 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.271618 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.271509 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.310538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.310488 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.312457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.312429 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:14:35.349526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.347996 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.509586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.509536 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.540018 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.539927 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:14:35.601413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.544920 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:35.601413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.545326 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:14:35.629416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.603372 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.629416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.604331 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.630988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.629848 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:14:35.630988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.629935 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:14:35.681785 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.679527 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.681785 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.679964 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:14:35.724559 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.724499 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.726560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.724795 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.827479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.799831 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.852415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.839021 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.852415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.839361 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.893451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.874513 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.920425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.898145 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:35.921051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.908974 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:14:35.921291 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.921272 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:35.921461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.921446 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:14:35.921621 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.921607 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:14:35.921755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.921743 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:14:35.934089 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.934054 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:14:35.972412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.971467 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 20:14:35.972412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.935755 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:35.972412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.934365 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:35.972725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.972441 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 20:14:35.972725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.972492 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:35.972725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.934677 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:35.972725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.972546 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 20:14:35.972725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.937581 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:35.972725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:35.972586 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:14:36.002578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.001521 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.002578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.001984 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:14:36.109497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.103341 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:14:36.156434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.147601 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.265448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.256105 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.345433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.321059 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.369053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.351138 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.393308 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.393275 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:14:36.526565 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.526520 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:14:36.614980 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.591699 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:36.615290 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.615259 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 20:14:36.615474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.595221 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.615771 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.603932 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.616058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.613983 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.616346 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.595178 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:14:36.642467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.595117 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:36.642719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.642688 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:14:36.758414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.758055 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:36.758414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.758114 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 20:14:36.840439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.815034 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:14:36.840749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.840709 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 20:14:36.857786 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.841626 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.857786 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.841947 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:14:36.857786 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.842158 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.857786 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.842344 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:36.927203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:36.927103 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.036049 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.035899 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:14:37.071415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.067990 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.170872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.170226 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.252772 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:14:37.222690 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:14:37.282493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.277592 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:14:37.321423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.308962 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:14:37.350511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.336178 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:14:37.375564 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.375517 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:14:37.412099 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.397791 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.555837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.555805 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:37.556089 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.556063 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 20:14:37.556335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.556313 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:14:37.558332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.558308 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:37.571738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.571702 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:14:37.601740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.601698 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.712414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.615028 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:14:37.712414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.646494 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.712414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.647421 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.712414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.648580 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.730941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.730905 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:37.731202 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.731174 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 20:14:37.731588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.731566 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:14:37.731745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.731723 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 20:14:37.853749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.853646 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.854421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.854363 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.854852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.854825 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.921657 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.921611 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:37.949411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:37.934034 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:38.070411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.038495 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:38.309049 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.301716 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:14:38.346952 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.346886 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:14:38.547414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.546752 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:38.547414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.546818 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 20:14:38.572286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.572081 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:38.572286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.572147 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:14:38.631451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.627619 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:14:38.766413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.737406 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:38.773804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.773756 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:14:38.795414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.788577 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:38.796732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.796692 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:38.804418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.798467 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:38.953557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.815447 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded" Nov 03 20:14:38.953797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.843444 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:14:38.954216 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.848436 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:38.954407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.954364 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 20:14:39.006801 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.869653 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:39.048473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.924526 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.048473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.954592 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:14:39.048473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.869601 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:14:39.048473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:38.849948 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.048473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.035981 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 20:14:39.078030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.077985 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:14:39.100436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.082898 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:14:39.223476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.218326 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:14:39.350297 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.349530 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:14:39.479847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.470480 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.612328 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.564303 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.612328 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.564365 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:14:39.612328 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.564463 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.612328 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.564488 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 20:14:39.641330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.631301 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.733476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.726706 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.740420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.733713 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 20:14:39.740668 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.739153 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.756566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.756522 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:14:39.756756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.756597 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 20:14:39.783524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.783475 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:14:39.828563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.826069 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.830269 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.829910 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:14:39.933423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.930772 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:39.963878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:39.963839 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:14:40.045332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.034936 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:14:40.051601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.051554 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:14:40.117410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.076608 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:40.117751 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.117698 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:40.142811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.142743 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:40.183957 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.181204 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:14:40.347990 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.334407 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:14:40.348253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.307037 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:40.495706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.491879 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:14:40.511890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.509947 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 03 20:14:40.557501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.552755 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:40.569599 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.569558 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.569922 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.569893 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:14:40.570103 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.569824 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.570231 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.570213 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 20:14:40.680471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.680000 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:40.772499 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.734520 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:14:40.809207 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.809162 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:14:40.809419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.809235 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 20:14:40.809419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.787684 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:40.987468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.987312 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:41.009414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:40.988201 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 03 20:14:41.015317 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.011592 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:41.018136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.015971 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:41.018136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.017136 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.018136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.017176 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 20:14:41.059440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.050856 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:41.085512 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.084940 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:41.098411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.085973 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:14:41.122055 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.119454 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:14:41.157057 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.157013 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:41.305469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.206082 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:41.564472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.561352 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:14:41.564472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.564001 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:14:41.570405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.565718 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.570405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.565771 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:14:41.570405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.566437 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.570405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.566471 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 03 20:14:41.582606 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.582562 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:41.583233 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.583209 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:14:41.637413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.627558 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:41.735411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.732583 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:14:41.789114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.789081 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.789419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.789360 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 20:14:41.856720 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.856631 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:41.856991 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.856944 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 20:14:41.858258 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.858225 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:14:41.862991 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.862959 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:42.026806 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:41.988952 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:42.062438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.062350 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:42.146479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.124805 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:14:42.164407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.161089 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:42.348741 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.348694 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:14:42.362414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.357736 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:14:42.562058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.562024 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 03 20:14:42.562336 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.562306 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:14:42.562867 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.562372 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:14:42.562867 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.562460 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:14:42.562867 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.562511 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:14:42.563610 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.562325 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 03 20:14:42.610548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.583534 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:42.610548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.586465 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:42.754906 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.753556 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:42.754906 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.753659 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 20:14:42.759411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.758048 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:14:42.759411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.758116 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 20:14:42.913445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.913293 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:14:42.915014 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.914910 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:14:42.916876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.916359 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:42.916876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.916448 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:14:42.916876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.916535 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:14:43.011347 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:42.988491 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:43.032489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.022677 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:43.068414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.062861 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:43.116610 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.116537 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:43.346469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.345756 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:14:43.377329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.377290 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:14:43.416091 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.413717 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:43.425839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.425802 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:43.501817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.487379 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:14:43.729461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.727924 4457 patch_prober.go:29] interesting pod/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Nov 03 20:14:43.729461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.727983 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Nov 03 20:14:43.729461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.728484 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Nov 03 20:14:43.729461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.728538 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Nov 03 20:14:43.857498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.856098 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:43.857498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.857356 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:43.878858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:43.878834 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:14:44.726990 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:44.726961 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:44.727505 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:44.727483 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:44.728299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:44.728275 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:14:44.730158 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:44.730139 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:44.730448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:44.730428 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:44.731202 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:44.731183 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:14:53.125421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.122462 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7"] Nov 03 20:14:53.191406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.188029 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" Nov 03 20:14:53.223813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.221639 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:14:53.223813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.221787 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:14:53.252417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.236852 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4jjq\" (UniqueName: \"kubernetes.io/projected/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-kube-api-access-m4jjq\") pod \"maas-api-revoke-sub-a2e6911f-hwfm7\" (UID: \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" Nov 03 20:14:53.253973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.253934 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2e6911f-hwfm7\" (UID: \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" Nov 03 20:14:53.257503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.257470 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2e6911f-hwfm7\" (UID: \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" Nov 03 20:14:53.409478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.384756 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4jjq\" (UniqueName: \"kubernetes.io/projected/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-kube-api-access-m4jjq\") pod \"maas-api-revoke-sub-a2e6911f-hwfm7\" (UID: \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" Nov 03 20:14:53.409478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.385001 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2e6911f-hwfm7\" (UID: \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" Nov 03 20:14:53.409478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.385049 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2e6911f-hwfm7\" (UID: \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" Nov 03 20:14:53.487150 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.487112 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2e6911f-hwfm7\" (UID: \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" Nov 03 20:14:53.690016 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.669924 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2e6911f-hwfm7\" (UID: \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" Nov 03 20:14:53.791380 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.719933 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:53.826414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.746411 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:53.826414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.765210 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4jjq\" (UniqueName: \"kubernetes.io/projected/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-kube-api-access-m4jjq\") pod \"maas-api-revoke-sub-a2e6911f-hwfm7\" (UID: \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" Nov 03 20:14:53.826414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.790125 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:53.852452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.849758 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:53.852452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.851353 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:53.902729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.902685 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" Nov 03 20:14:53.936423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.917246 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7"] Nov 03 20:14:54.061439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.041540 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.061439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.938007 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.061439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.938055 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.061439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:53.985826 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.061439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.052886 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.074502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.074448 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.373900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.373844 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.377547 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.374529 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:14:54.457543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.457480 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:14:54.733675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.727921 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:14:54.805414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.797477 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:14:54.805414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.798081 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:14:54.805414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.804322 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:14:54.805414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.804954 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:14:54.822500 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.822451 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.823121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.823096 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:14:54.853742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.853702 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.854371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.854347 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:14:54.863407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.854550 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:54.867568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:54.867545 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:14:55.086210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.076175 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.086210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.076931 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:14:55.086210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.078231 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.086210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.078297 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.086210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.078312 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:14:55.086210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.078563 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:14:55.086210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.078598 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:14:55.086210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.078645 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:14:55.086210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.079965 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.112295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.092961 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:14:55.112295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.093372 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:14:55.165416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.157500 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Nov 03 20:14:55.305931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.305892 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:14:55.306254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.306200 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" podUID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.457466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.419979 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:14:55.457466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.420043 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" podUID="5d8658c4-9961-4acd-8180-20128324c476" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.457466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.420317 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:14:55.457466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.420965 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.457466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.431514 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:14:55.457466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.431591 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:14:55.460551 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.460513 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:14:55.588940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.568949 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:14:55.671412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.636410 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:14:55.749605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.748505 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.752723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.752684 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.799678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.790557 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:14:55.801349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.801307 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:14:55.829546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.829482 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.832445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.831803 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:55.904609 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.904546 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.061476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:55.905080 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.086414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:56.081249 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.094050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:56.094007 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.117428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:56.103972 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.246413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:56.232379 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.248085 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:14:56.238143 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94fdcc8_e4e1_4384_beb7_aad82f1bed5b.slice/crio-18ebd38d583ef320558bf53dbb25056c42d018298d8707d4639eac9d36f3d600 WatchSource:0}: Error finding container 18ebd38d583ef320558bf53dbb25056c42d018298d8707d4639eac9d36f3d600: Status 404 returned error can't find the container with id 18ebd38d583ef320558bf53dbb25056c42d018298d8707d4639eac9d36f3d600 Nov 03 20:14:56.270894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:56.266839 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:14:56.454420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:56.430378 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.976492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:56.431916 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:56.976492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:56.630479 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:14:56.976492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:56.784014 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:14:56.976492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:56.848920 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:14:56.976492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:56.849407 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:14:56.976492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:56.849704 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 03 20:14:57.038730 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.035731 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.117411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.110424 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.117411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.110821 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:14:57.117411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.114641 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.117411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.115464 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.257877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.248493 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.308421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.302904 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.352583 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.352533 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.481413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.475684 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" event={"ID":"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b","Type":"ContainerStarted","Data":"18ebd38d583ef320558bf53dbb25056c42d018298d8707d4639eac9d36f3d600"} Nov 03 20:14:57.611619 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.611350 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7"] Nov 03 20:14:57.684852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.684808 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.717762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.717492 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.843479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.839511 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:14:57.843479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.839924 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.843479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.840208 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:14:57.843479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.840490 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:14:57.868401 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.858447 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.868401 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.858797 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:57.868401 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:57.859058 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:14:58.041231 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.041184 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:14:58.082605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.082552 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:14:58.082937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.082885 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.112523 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.112475 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.112903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.112777 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.114656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.114635 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:14:58.123819 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.116060 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.142468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.125079 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:14:58.249545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.249489 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.310449 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.304493 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.711656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.711547 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.775094 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.726611 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.854023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.852560 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.855662 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.854528 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.856954 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.855855 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:14:58.858917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.857127 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:14:58.858917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.857598 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:14:58.877475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.869222 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:58.877475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.870606 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 03 20:14:58.877475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.870809 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:14:58.965173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.948889 4457 generic.go:356] "Generic (PLEG): container finished" podID="b94fdcc8-e4e1-4384-beb7-aad82f1bed5b" containerID="c9f3ac61b0aa4cbcb9fd29e0ce0c68db6c1815da90f98b2c65c42d6832e4554e" exitCode=0 Nov 03 20:14:58.965173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:58.949033 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" event={"ID":"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b","Type":"ContainerDied","Data":"c9f3ac61b0aa4cbcb9fd29e0ce0c68db6c1815da90f98b2c65c42d6832e4554e"} Nov 03 20:14:59.049259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.042615 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:14:59.114799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.114745 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:59.151024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.150979 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 03 20:14:59.165806 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.156870 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:59.165806 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.159727 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:14:59.256845 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.255328 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:59.317430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.312491 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:14:59.464589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.464536 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:14:59.464792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.464616 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:59.847459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.846892 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:14:59.871413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.854109 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:59.871413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.861540 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:14:59.871413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.861618 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:59.871413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.861541 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:59.896412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.883487 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:14:59.896412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.884018 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:14:59.896412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.884279 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 03 20:14:59.964406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:14:59.961244 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:00.050936 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.050898 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:00.113446 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.112232 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:15:00.137144 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.137090 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:00.158410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.157488 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:00.158410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.158244 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 03 20:15:00.160126 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.160096 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 03 20:15:00.206406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.201594 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:00.227838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.223605 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:15:00.227838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.223681 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:15:00.247457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.247077 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9"] Nov 03 20:15:00.287581 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.287549 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9"] Nov 03 20:15:00.292250 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.292208 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" Nov 03 20:15:00.294134 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.294116 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:15:00.294291 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.294281 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:15:00.294435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.294425 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:15:00.294545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.294536 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:15:00.294643 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.294633 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:15:00.294735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.294724 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9"] Nov 03 20:15:00.294862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.294852 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:15:00.294945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.294936 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9"] Nov 03 20:15:00.295030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.295021 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg"] Nov 03 20:15:00.311405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.311042 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" Nov 03 20:15:00.315682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.314564 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:15:00.315682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.314607 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:15:00.315682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.314622 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg"] Nov 03 20:15:00.315682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.314661 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:15:00.315682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.314693 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:15:00.315682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.315248 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" Nov 03 20:15:00.322773 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.322708 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:15:00.429494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.426139 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc9e13f-05f6-4a9e-a15c-034c176b4695-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9\" (UID: \"7cc9e13f-05f6-4a9e-a15c-034c176b4695\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" Nov 03 20:15:00.429494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.426442 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg\" (UID: \"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" Nov 03 20:15:00.429494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.426587 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmfzb\" (UniqueName: \"kubernetes.io/projected/7cc9e13f-05f6-4a9e-a15c-034c176b4695-kube-api-access-bmfzb\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9\" (UID: \"7cc9e13f-05f6-4a9e-a15c-034c176b4695\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" Nov 03 20:15:00.429494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.426958 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfprv\" (UniqueName: \"kubernetes.io/projected/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-kube-api-access-hfprv\") pod \"maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9\" (UID: \"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" Nov 03 20:15:00.429494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.426987 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfxnd\" (UniqueName: \"kubernetes.io/projected/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-kube-api-access-mfxnd\") pod \"maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg\" (UID: \"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" Nov 03 20:15:00.429494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.427100 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9\" (UID: \"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" Nov 03 20:15:00.509988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.509928 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" Nov 03 20:15:00.528555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.528521 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmfzb\" (UniqueName: \"kubernetes.io/projected/7cc9e13f-05f6-4a9e-a15c-034c176b4695-kube-api-access-bmfzb\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9\" (UID: \"7cc9e13f-05f6-4a9e-a15c-034c176b4695\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" Nov 03 20:15:00.528706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.528681 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfprv\" (UniqueName: \"kubernetes.io/projected/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-kube-api-access-hfprv\") pod \"maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9\" (UID: \"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" Nov 03 20:15:00.528754 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.528705 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfxnd\" (UniqueName: \"kubernetes.io/projected/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-kube-api-access-mfxnd\") pod \"maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg\" (UID: \"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" Nov 03 20:15:00.528754 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.528741 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9\" (UID: \"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" Nov 03 20:15:00.528851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.528839 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc9e13f-05f6-4a9e-a15c-034c176b4695-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9\" (UID: \"7cc9e13f-05f6-4a9e-a15c-034c176b4695\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" Nov 03 20:15:00.528909 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.528894 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg\" (UID: \"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" Nov 03 20:15:00.529549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.529528 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9\" (UID: \"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" Nov 03 20:15:00.529986 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.529749 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc9e13f-05f6-4a9e-a15c-034c176b4695-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9\" (UID: \"7cc9e13f-05f6-4a9e-a15c-034c176b4695\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" Nov 03 20:15:00.529986 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.529755 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg\" (UID: \"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" Nov 03 20:15:00.542310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.542265 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt"] Nov 03 20:15:00.542973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.542953 4457 cpu_manager.go:414] "containerMap: removing container" podUID="b94fdcc8-e4e1-4384-beb7-aad82f1bed5b" containerName="revoke-keys" Nov 03 20:15:00.542973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.542969 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b94fdcc8-e4e1-4384-beb7-aad82f1bed5b" containerName="revoke-keys" Nov 03 20:15:00.543145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.543105 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="b94fdcc8-e4e1-4384-beb7-aad82f1bed5b" containerName="revoke-keys" Nov 03 20:15:00.550038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.550012 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" Nov 03 20:15:00.580124 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.579664 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt"] Nov 03 20:15:00.596988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.585346 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfxnd\" (UniqueName: \"kubernetes.io/projected/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-kube-api-access-mfxnd\") pod \"maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg\" (UID: \"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" Nov 03 20:15:00.596988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.586446 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfprv\" (UniqueName: \"kubernetes.io/projected/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-kube-api-access-hfprv\") pod \"maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9\" (UID: \"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" Nov 03 20:15:00.596988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.587101 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmfzb\" (UniqueName: \"kubernetes.io/projected/7cc9e13f-05f6-4a9e-a15c-034c176b4695-kube-api-access-bmfzb\") pod \"maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9\" (UID: \"7cc9e13f-05f6-4a9e-a15c-034c176b4695\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" Nov 03 20:15:00.630218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.630138 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-kube-api-access-m4jjq\" (UniqueName: \"kubernetes.io/projected/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-kube-api-access-m4jjq\") pod \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\" (UID: \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\") " Nov 03 20:15:00.630351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.630313 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-xks-ca\") pod \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\" (UID: \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\") " Nov 03 20:15:00.630433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.630354 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-openshift-ca\") pod \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\" (UID: \"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b\") " Nov 03 20:15:00.630778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.630738 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-openshift-ca" pod "b94fdcc8-e4e1-4384-beb7-aad82f1bed5b" (UID: "b94fdcc8-e4e1-4384-beb7-aad82f1bed5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:00.630890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.630826 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f782b7-a701-419b-9326-7bc84605807f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8a81f6-wlwpt\" (UID: \"81f782b7-a701-419b-9326-7bc84605807f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" Nov 03 20:15:00.633380 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.633348 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f782b7-a701-419b-9326-7bc84605807f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8a81f6-wlwpt\" (UID: \"81f782b7-a701-419b-9326-7bc84605807f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" Nov 03 20:15:00.633568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.633550 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zg6f\" (UniqueName: \"kubernetes.io/projected/81f782b7-a701-419b-9326-7bc84605807f-kube-api-access-2zg6f\") pod \"maas-api-revoke-sub-cf8a81f6-wlwpt\" (UID: \"81f782b7-a701-419b-9326-7bc84605807f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" Nov 03 20:15:00.633654 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.633640 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:00.633974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.633953 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-xks-ca" pod "b94fdcc8-e4e1-4384-beb7-aad82f1bed5b" (UID: "b94fdcc8-e4e1-4384-beb7-aad82f1bed5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:00.634124 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.634106 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-kube-api-access-m4jjq" pod "b94fdcc8-e4e1-4384-beb7-aad82f1bed5b" (UID: "b94fdcc8-e4e1-4384-beb7-aad82f1bed5b"). InnerVolumeSpecName "kube-api-access-m4jjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:00.641829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.641798 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" Nov 03 20:15:00.645756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.645734 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" Nov 03 20:15:00.652872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.652846 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" Nov 03 20:15:00.738415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.738326 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zg6f\" (UniqueName: \"kubernetes.io/projected/81f782b7-a701-419b-9326-7bc84605807f-kube-api-access-2zg6f\") pod \"maas-api-revoke-sub-cf8a81f6-wlwpt\" (UID: \"81f782b7-a701-419b-9326-7bc84605807f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" Nov 03 20:15:00.738774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.738679 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f782b7-a701-419b-9326-7bc84605807f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8a81f6-wlwpt\" (UID: \"81f782b7-a701-419b-9326-7bc84605807f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" Nov 03 20:15:00.739125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.739078 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f782b7-a701-419b-9326-7bc84605807f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8a81f6-wlwpt\" (UID: \"81f782b7-a701-419b-9326-7bc84605807f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" Nov 03 20:15:00.739452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.739403 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4jjq\" (UniqueName: \"kubernetes.io/projected/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-kube-api-access-m4jjq\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:00.739452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.739431 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:00.739619 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.739603 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f782b7-a701-419b-9326-7bc84605807f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8a81f6-wlwpt\" (UID: \"81f782b7-a701-419b-9326-7bc84605807f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" Nov 03 20:15:00.742169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.742122 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f782b7-a701-419b-9326-7bc84605807f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8a81f6-wlwpt\" (UID: \"81f782b7-a701-419b-9326-7bc84605807f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" Nov 03 20:15:00.780219 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.780139 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zg6f\" (UniqueName: \"kubernetes.io/projected/81f782b7-a701-419b-9326-7bc84605807f-kube-api-access-2zg6f\") pod \"maas-api-revoke-sub-cf8a81f6-wlwpt\" (UID: \"81f782b7-a701-419b-9326-7bc84605807f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" Nov 03 20:15:00.894652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.883463 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn"] Nov 03 20:15:00.894652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.886246 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" Nov 03 20:15:00.908528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.908504 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn"] Nov 03 20:15:00.908789 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:00.908775 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" Nov 03 20:15:01.016438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.016400 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9"] Nov 03 20:15:01.058053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.058019 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kft9k\" (UniqueName: \"kubernetes.io/projected/17465221-8104-4091-8e04-0e89431e264d-kube-api-access-kft9k\") pod \"maas-api-revoke-sub-06abad0c-9vzzn\" (UID: \"17465221-8104-4091-8e04-0e89431e264d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" Nov 03 20:15:01.060437 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.060247 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17465221-8104-4091-8e04-0e89431e264d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06abad0c-9vzzn\" (UID: \"17465221-8104-4091-8e04-0e89431e264d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" Nov 03 20:15:01.060821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.060798 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17465221-8104-4091-8e04-0e89431e264d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06abad0c-9vzzn\" (UID: \"17465221-8104-4091-8e04-0e89431e264d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" Nov 03 20:15:01.063072 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.063024 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" event={"ID":"b94fdcc8-e4e1-4384-beb7-aad82f1bed5b","Type":"ContainerDied","Data":"18ebd38d583ef320558bf53dbb25056c42d018298d8707d4639eac9d36f3d600"} Nov 03 20:15:01.063469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.063453 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18ebd38d583ef320558bf53dbb25056c42d018298d8707d4639eac9d36f3d600" Nov 03 20:15:01.063623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.063377 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7" Nov 03 20:15:01.072065 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:15:01.072019 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6fe6d5_4a7f_46a1_b19c_f3ae1a17665c.slice/crio-0086cc6fb1edc1e843d8503edc319af25cc509d5967139182c741d33dac32a85 WatchSource:0}: Error finding container 0086cc6fb1edc1e843d8503edc319af25cc509d5967139182c741d33dac32a85: Status 404 returned error can't find the container with id 0086cc6fb1edc1e843d8503edc319af25cc509d5967139182c741d33dac32a85 Nov 03 20:15:01.165403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.163958 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17465221-8104-4091-8e04-0e89431e264d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06abad0c-9vzzn\" (UID: \"17465221-8104-4091-8e04-0e89431e264d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" Nov 03 20:15:01.165403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.164179 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kft9k\" (UniqueName: \"kubernetes.io/projected/17465221-8104-4091-8e04-0e89431e264d-kube-api-access-kft9k\") pod \"maas-api-revoke-sub-06abad0c-9vzzn\" (UID: \"17465221-8104-4091-8e04-0e89431e264d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" Nov 03 20:15:01.165403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.164279 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17465221-8104-4091-8e04-0e89431e264d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06abad0c-9vzzn\" (UID: \"17465221-8104-4091-8e04-0e89431e264d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" Nov 03 20:15:01.165403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.165286 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17465221-8104-4091-8e04-0e89431e264d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06abad0c-9vzzn\" (UID: \"17465221-8104-4091-8e04-0e89431e264d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" Nov 03 20:15:01.224769 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.224741 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9"] Nov 03 20:15:01.244858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.244834 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg"] Nov 03 20:15:01.257001 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.256977 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17465221-8104-4091-8e04-0e89431e264d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06abad0c-9vzzn\" (UID: \"17465221-8104-4091-8e04-0e89431e264d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" Nov 03 20:15:01.257457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.257431 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kft9k\" (UniqueName: \"kubernetes.io/projected/17465221-8104-4091-8e04-0e89431e264d-kube-api-access-kft9k\") pod \"maas-api-revoke-sub-06abad0c-9vzzn\" (UID: \"17465221-8104-4091-8e04-0e89431e264d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" Nov 03 20:15:01.259590 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:15:01.259550 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc9e13f_05f6_4a9e_a15c_034c176b4695.slice/crio-67ffc5f97ace287f02ad1e272033ba03babbb5f0b6c7825f6218ade3163ec6f6 WatchSource:0}: Error finding container 67ffc5f97ace287f02ad1e272033ba03babbb5f0b6c7825f6218ade3163ec6f6: Status 404 returned error can't find the container with id 67ffc5f97ace287f02ad1e272033ba03babbb5f0b6c7825f6218ade3163ec6f6 Nov 03 20:15:01.275712 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:15:01.275679 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f782b7_a701_419b_9326_7bc84605807f.slice/crio-0c74b8de4242bdebdcaed39aa4e57a3f876804e49426dba32b4d95f726dcc2c0 WatchSource:0}: Error finding container 0c74b8de4242bdebdcaed39aa4e57a3f876804e49426dba32b4d95f726dcc2c0: Status 404 returned error can't find the container with id 0c74b8de4242bdebdcaed39aa4e57a3f876804e49426dba32b4d95f726dcc2c0 Nov 03 20:15:01.276327 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.276302 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt"] Nov 03 20:15:01.535822 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.535791 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" Nov 03 20:15:01.923537 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:15:01.923497 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17465221_8104_4091_8e04_0e89431e264d.slice/crio-274fe389ad22b74819b9eb21c2197526956c0fa355a3af700109eae4dd484d55 WatchSource:0}: Error finding container 274fe389ad22b74819b9eb21c2197526956c0fa355a3af700109eae4dd484d55: Status 404 returned error can't find the container with id 274fe389ad22b74819b9eb21c2197526956c0fa355a3af700109eae4dd484d55 Nov 03 20:15:01.925559 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:01.925529 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn"] Nov 03 20:15:02.074836 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:02.074785 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" event={"ID":"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a","Type":"ContainerStarted","Data":"b2f3b6281b172a197f3eb927cbeaa0db65822a0582320bc6df9f371839b6db9e"} Nov 03 20:15:02.075287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:02.074902 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" event={"ID":"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a","Type":"ContainerStarted","Data":"4a74ace746908bf20ae9dcf75b6b7a5403da5cbb960378123bc364a0f18c1581"} Nov 03 20:15:02.076843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:02.076808 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" event={"ID":"7cc9e13f-05f6-4a9e-a15c-034c176b4695","Type":"ContainerStarted","Data":"660bd6408a00067e1295aa3f6d4a6b2526d7623a3ebbbfc1b07f03f361a80899"} Nov 03 20:15:02.077030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:02.076879 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" event={"ID":"7cc9e13f-05f6-4a9e-a15c-034c176b4695","Type":"ContainerStarted","Data":"67ffc5f97ace287f02ad1e272033ba03babbb5f0b6c7825f6218ade3163ec6f6"} Nov 03 20:15:02.079119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:02.079089 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" event={"ID":"81f782b7-a701-419b-9326-7bc84605807f","Type":"ContainerStarted","Data":"64ba715241566e4f627d58f10820607d2b8029257345d85dff95f2f6fe969df1"} Nov 03 20:15:02.079212 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:02.079141 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" event={"ID":"81f782b7-a701-419b-9326-7bc84605807f","Type":"ContainerStarted","Data":"0c74b8de4242bdebdcaed39aa4e57a3f876804e49426dba32b4d95f726dcc2c0"} Nov 03 20:15:02.080699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:02.080674 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" event={"ID":"17465221-8104-4091-8e04-0e89431e264d","Type":"ContainerStarted","Data":"274fe389ad22b74819b9eb21c2197526956c0fa355a3af700109eae4dd484d55"} Nov 03 20:15:02.082451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:02.082425 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" event={"ID":"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c","Type":"ContainerStarted","Data":"81db0458f369c580ef1864275b9065cc36e4fd01750841f36bb847bebe700bb1"} Nov 03 20:15:02.082539 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:02.082473 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" event={"ID":"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c","Type":"ContainerStarted","Data":"0086cc6fb1edc1e843d8503edc319af25cc509d5967139182c741d33dac32a85"} Nov 03 20:15:02.093632 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:02.093581 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" podStartSLOduration=2.093569802 podStartE2EDuration="2.093569802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.091766414 +0000 UTC m=+1080.467363162" watchObservedRunningTime="2026-11-03 20:15:02.093569802 +0000 UTC m=+1080.469166548" Nov 03 20:15:02.113764 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:02.113703 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" podStartSLOduration=2.113684766 podStartE2EDuration="2.113684766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.109667544 +0000 UTC m=+1080.485264292" watchObservedRunningTime="2026-11-03 20:15:02.113684766 +0000 UTC m=+1080.489281515" Nov 03 20:15:02.129455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:02.129369 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" podStartSLOduration=2.1293518049999998 podStartE2EDuration="2.129351805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.12808065 +0000 UTC m=+1080.503677408" watchObservedRunningTime="2026-11-03 20:15:02.129351805 +0000 UTC m=+1080.504948552" Nov 03 20:15:02.150658 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:02.150606 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" podStartSLOduration=2.150593223 podStartE2EDuration="2.150593223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:02.149977576 +0000 UTC m=+1080.525574323" watchObservedRunningTime="2026-11-03 20:15:02.150593223 +0000 UTC m=+1080.526189970" Nov 03 20:15:03.099563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:03.099523 4457 generic.go:356] "Generic (PLEG): container finished" podID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" containerID="660bd6408a00067e1295aa3f6d4a6b2526d7623a3ebbbfc1b07f03f361a80899" exitCode=60 Nov 03 20:15:03.100013 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:03.099611 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" event={"ID":"7cc9e13f-05f6-4a9e-a15c-034c176b4695","Type":"ContainerDied","Data":"660bd6408a00067e1295aa3f6d4a6b2526d7623a3ebbbfc1b07f03f361a80899"} Nov 03 20:15:03.100013 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:03.099964 4457 scope.go:122] "RemoveContainer" containerID="660bd6408a00067e1295aa3f6d4a6b2526d7623a3ebbbfc1b07f03f361a80899" Nov 03 20:15:03.102051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:03.102027 4457 generic.go:356] "Generic (PLEG): container finished" podID="81f782b7-a701-419b-9326-7bc84605807f" containerID="64ba715241566e4f627d58f10820607d2b8029257345d85dff95f2f6fe969df1" exitCode=0 Nov 03 20:15:03.102165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:03.102095 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" event={"ID":"81f782b7-a701-419b-9326-7bc84605807f","Type":"ContainerDied","Data":"64ba715241566e4f627d58f10820607d2b8029257345d85dff95f2f6fe969df1"} Nov 03 20:15:03.105224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:03.105199 4457 generic.go:356] "Generic (PLEG): container finished" podID="17465221-8104-4091-8e04-0e89431e264d" containerID="33cdcb68dd54521a2f7a158b2205104306cacb53fe276a6bc8568673180aadee" exitCode=0 Nov 03 20:15:03.105332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:03.105272 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" event={"ID":"17465221-8104-4091-8e04-0e89431e264d","Type":"ContainerDied","Data":"33cdcb68dd54521a2f7a158b2205104306cacb53fe276a6bc8568673180aadee"} Nov 03 20:15:03.107517 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:03.107495 4457 generic.go:356] "Generic (PLEG): container finished" podID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" containerID="81db0458f369c580ef1864275b9065cc36e4fd01750841f36bb847bebe700bb1" exitCode=60 Nov 03 20:15:03.107758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:03.107734 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" event={"ID":"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c","Type":"ContainerDied","Data":"81db0458f369c580ef1864275b9065cc36e4fd01750841f36bb847bebe700bb1"} Nov 03 20:15:03.107898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:03.107812 4457 scope.go:122] "RemoveContainer" containerID="81db0458f369c580ef1864275b9065cc36e4fd01750841f36bb847bebe700bb1" Nov 03 20:15:03.109840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:03.109672 4457 generic.go:356] "Generic (PLEG): container finished" podID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" containerID="b2f3b6281b172a197f3eb927cbeaa0db65822a0582320bc6df9f371839b6db9e" exitCode=60 Nov 03 20:15:03.109840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:03.109700 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" event={"ID":"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a","Type":"ContainerDied","Data":"b2f3b6281b172a197f3eb927cbeaa0db65822a0582320bc6df9f371839b6db9e"} Nov 03 20:15:03.109964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:03.109933 4457 scope.go:122] "RemoveContainer" containerID="b2f3b6281b172a197f3eb927cbeaa0db65822a0582320bc6df9f371839b6db9e" Nov 03 20:15:04.121407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.120863 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" event={"ID":"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c","Type":"ContainerStarted","Data":"a0cb21909def09780587d124764de283b0cdfa22ccb5853a5ac0284259150964"} Nov 03 20:15:04.123709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.123672 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" event={"ID":"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a","Type":"ContainerStarted","Data":"fd2656867f7475afb332b3b88da09fbda88752f359491b2e37640fcb48a9d3f0"} Nov 03 20:15:04.126629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.126579 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" event={"ID":"7cc9e13f-05f6-4a9e-a15c-034c176b4695","Type":"ContainerStarted","Data":"b9f269b8052724c32ec59cceda48bc4c3d556d05e22c29a8635642449ba40f84"} Nov 03 20:15:04.522117 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.522093 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" Nov 03 20:15:04.528411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.528373 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" Nov 03 20:15:04.646412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.646305 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81f782b7-a701-419b-9326-7bc84605807f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f782b7-a701-419b-9326-7bc84605807f-maas-api-xks-ca\") pod \"81f782b7-a701-419b-9326-7bc84605807f\" (UID: \"81f782b7-a701-419b-9326-7bc84605807f\") " Nov 03 20:15:04.646589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.646457 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81f782b7-a701-419b-9326-7bc84605807f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f782b7-a701-419b-9326-7bc84605807f-maas-api-openshift-ca\") pod \"81f782b7-a701-419b-9326-7bc84605807f\" (UID: \"81f782b7-a701-419b-9326-7bc84605807f\") " Nov 03 20:15:04.646589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.646526 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81f782b7-a701-419b-9326-7bc84605807f-kube-api-access-2zg6f\" (UniqueName: \"kubernetes.io/projected/81f782b7-a701-419b-9326-7bc84605807f-kube-api-access-2zg6f\") pod \"81f782b7-a701-419b-9326-7bc84605807f\" (UID: \"81f782b7-a701-419b-9326-7bc84605807f\") " Nov 03 20:15:04.646589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.646552 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17465221-8104-4091-8e04-0e89431e264d-kube-api-access-kft9k\" (UniqueName: \"kubernetes.io/projected/17465221-8104-4091-8e04-0e89431e264d-kube-api-access-kft9k\") pod \"17465221-8104-4091-8e04-0e89431e264d\" (UID: \"17465221-8104-4091-8e04-0e89431e264d\") " Nov 03 20:15:04.646775 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.646744 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17465221-8104-4091-8e04-0e89431e264d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17465221-8104-4091-8e04-0e89431e264d-maas-api-openshift-ca\") pod \"17465221-8104-4091-8e04-0e89431e264d\" (UID: \"17465221-8104-4091-8e04-0e89431e264d\") " Nov 03 20:15:04.646834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.646811 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17465221-8104-4091-8e04-0e89431e264d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17465221-8104-4091-8e04-0e89431e264d-maas-api-xks-ca\") pod \"17465221-8104-4091-8e04-0e89431e264d\" (UID: \"17465221-8104-4091-8e04-0e89431e264d\") " Nov 03 20:15:04.646834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.646817 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f782b7-a701-419b-9326-7bc84605807f-maas-api-openshift-ca" pod "81f782b7-a701-419b-9326-7bc84605807f" (UID: "81f782b7-a701-419b-9326-7bc84605807f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:04.647166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.647135 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17465221-8104-4091-8e04-0e89431e264d-maas-api-openshift-ca" pod "17465221-8104-4091-8e04-0e89431e264d" (UID: "17465221-8104-4091-8e04-0e89431e264d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:04.647717 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.647693 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81f782b7-a701-419b-9326-7bc84605807f-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:04.647778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.647726 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17465221-8104-4091-8e04-0e89431e264d-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:04.648705 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.648682 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f782b7-a701-419b-9326-7bc84605807f-kube-api-access-2zg6f" pod "81f782b7-a701-419b-9326-7bc84605807f" (UID: "81f782b7-a701-419b-9326-7bc84605807f"). InnerVolumeSpecName "kube-api-access-2zg6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:04.648796 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.648715 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81f782b7-a701-419b-9326-7bc84605807f-maas-api-xks-ca" pod "81f782b7-a701-419b-9326-7bc84605807f" (UID: "81f782b7-a701-419b-9326-7bc84605807f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:04.649111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.649089 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17465221-8104-4091-8e04-0e89431e264d-maas-api-xks-ca" pod "17465221-8104-4091-8e04-0e89431e264d" (UID: "17465221-8104-4091-8e04-0e89431e264d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:04.649111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.649097 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17465221-8104-4091-8e04-0e89431e264d-kube-api-access-kft9k" pod "17465221-8104-4091-8e04-0e89431e264d" (UID: "17465221-8104-4091-8e04-0e89431e264d"). InnerVolumeSpecName "kube-api-access-kft9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:04.749351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.749316 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81f782b7-a701-419b-9326-7bc84605807f-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:04.749351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.749347 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zg6f\" (UniqueName: \"kubernetes.io/projected/81f782b7-a701-419b-9326-7bc84605807f-kube-api-access-2zg6f\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:04.749351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.749357 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kft9k\" (UniqueName: \"kubernetes.io/projected/17465221-8104-4091-8e04-0e89431e264d-kube-api-access-kft9k\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:04.749592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:04.749367 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17465221-8104-4091-8e04-0e89431e264d-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:05.135678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.135633 4457 generic.go:356] "Generic (PLEG): container finished" podID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" containerID="fd2656867f7475afb332b3b88da09fbda88752f359491b2e37640fcb48a9d3f0" exitCode=60 Nov 03 20:15:05.136121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.135709 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" event={"ID":"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a","Type":"ContainerDied","Data":"fd2656867f7475afb332b3b88da09fbda88752f359491b2e37640fcb48a9d3f0"} Nov 03 20:15:05.136121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.135777 4457 scope.go:122] "RemoveContainer" containerID="b2f3b6281b172a197f3eb927cbeaa0db65822a0582320bc6df9f371839b6db9e" Nov 03 20:15:05.136121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.136098 4457 scope.go:122] "RemoveContainer" containerID="fd2656867f7475afb332b3b88da09fbda88752f359491b2e37640fcb48a9d3f0" Nov 03 20:15:05.136579 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:05.136552 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg_odh-ai-gateway-infra(809babf5-bc1f-4a26-8fbd-67f1a5d0f11a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" podUID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" Nov 03 20:15:05.137982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.137962 4457 generic.go:356] "Generic (PLEG): container finished" podID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" containerID="b9f269b8052724c32ec59cceda48bc4c3d556d05e22c29a8635642449ba40f84" exitCode=60 Nov 03 20:15:05.138129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.138100 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" event={"ID":"7cc9e13f-05f6-4a9e-a15c-034c176b4695","Type":"ContainerDied","Data":"b9f269b8052724c32ec59cceda48bc4c3d556d05e22c29a8635642449ba40f84"} Nov 03 20:15:05.138453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.138432 4457 scope.go:122] "RemoveContainer" containerID="b9f269b8052724c32ec59cceda48bc4c3d556d05e22c29a8635642449ba40f84" Nov 03 20:15:05.138830 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:05.138807 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9_odh-ai-gateway-infra(7cc9e13f-05f6-4a9e-a15c-034c176b4695)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" podUID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" Nov 03 20:15:05.139570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.139541 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" Nov 03 20:15:05.139668 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.139587 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt" event={"ID":"81f782b7-a701-419b-9326-7bc84605807f","Type":"ContainerDied","Data":"0c74b8de4242bdebdcaed39aa4e57a3f876804e49426dba32b4d95f726dcc2c0"} Nov 03 20:15:05.139668 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.139630 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c74b8de4242bdebdcaed39aa4e57a3f876804e49426dba32b4d95f726dcc2c0" Nov 03 20:15:05.141267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.141251 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" Nov 03 20:15:05.141360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.141273 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn" event={"ID":"17465221-8104-4091-8e04-0e89431e264d","Type":"ContainerDied","Data":"274fe389ad22b74819b9eb21c2197526956c0fa355a3af700109eae4dd484d55"} Nov 03 20:15:05.141360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.141319 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="274fe389ad22b74819b9eb21c2197526956c0fa355a3af700109eae4dd484d55" Nov 03 20:15:05.144883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.144858 4457 generic.go:356] "Generic (PLEG): container finished" podID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" containerID="a0cb21909def09780587d124764de283b0cdfa22ccb5853a5ac0284259150964" exitCode=60 Nov 03 20:15:05.144968 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.144894 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" event={"ID":"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c","Type":"ContainerDied","Data":"a0cb21909def09780587d124764de283b0cdfa22ccb5853a5ac0284259150964"} Nov 03 20:15:05.145315 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.145296 4457 scope.go:122] "RemoveContainer" containerID="a0cb21909def09780587d124764de283b0cdfa22ccb5853a5ac0284259150964" Nov 03 20:15:05.145643 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:05.145623 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9_odh-ai-gateway-infra(dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" podUID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" Nov 03 20:15:05.152754 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.152737 4457 scope.go:122] "RemoveContainer" containerID="660bd6408a00067e1295aa3f6d4a6b2526d7623a3ebbbfc1b07f03f361a80899" Nov 03 20:15:05.171594 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:05.171479 4457 scope.go:122] "RemoveContainer" containerID="81db0458f369c580ef1864275b9065cc36e4fd01750841f36bb847bebe700bb1" Nov 03 20:15:06.076593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.076562 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l"] Nov 03 20:15:06.077236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.077221 4457 cpu_manager.go:414] "containerMap: removing container" podUID="81f782b7-a701-419b-9326-7bc84605807f" containerName="revoke-keys" Nov 03 20:15:06.077275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.077238 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81f782b7-a701-419b-9326-7bc84605807f" containerName="revoke-keys" Nov 03 20:15:06.077306 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.077276 4457 cpu_manager.go:414] "containerMap: removing container" podUID="17465221-8104-4091-8e04-0e89431e264d" containerName="revoke-keys" Nov 03 20:15:06.077306 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.077282 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17465221-8104-4091-8e04-0e89431e264d" containerName="revoke-keys" Nov 03 20:15:06.077419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.077399 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="81f782b7-a701-419b-9326-7bc84605807f" containerName="revoke-keys" Nov 03 20:15:06.077464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.077426 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="17465221-8104-4091-8e04-0e89431e264d" containerName="revoke-keys" Nov 03 20:15:06.082547 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.082526 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" Nov 03 20:15:06.086769 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.086749 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:06.092028 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.091996 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l"] Nov 03 20:15:06.163819 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.163784 4457 scope.go:122] "RemoveContainer" containerID="fd2656867f7475afb332b3b88da09fbda88752f359491b2e37640fcb48a9d3f0" Nov 03 20:15:06.164429 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:06.164372 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg_odh-ai-gateway-infra(809babf5-bc1f-4a26-8fbd-67f1a5d0f11a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" podUID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" Nov 03 20:15:06.166783 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.166629 4457 scope.go:122] "RemoveContainer" containerID="b9f269b8052724c32ec59cceda48bc4c3d556d05e22c29a8635642449ba40f84" Nov 03 20:15:06.166783 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:06.166686 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:15:06.167006 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:06.166989 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9_odh-ai-gateway-infra(7cc9e13f-05f6-4a9e-a15c-034c176b4695)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" podUID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" Nov 03 20:15:06.168629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.168608 4457 scope.go:122] "RemoveContainer" containerID="a0cb21909def09780587d124764de283b0cdfa22ccb5853a5ac0284259150964" Nov 03 20:15:06.168964 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:06.168938 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9_odh-ai-gateway-infra(dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" podUID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" Nov 03 20:15:06.172276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.172253 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65ae418c-6xl2l\" (UID: \"acab3741-a3ca-415d-b8e4-97143771efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" Nov 03 20:15:06.172367 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.172331 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k62c\" (UniqueName: \"kubernetes.io/projected/acab3741-a3ca-415d-b8e4-97143771efb9-kube-api-access-4k62c\") pod \"maas-api-revoke-sub-65ae418c-6xl2l\" (UID: \"acab3741-a3ca-415d-b8e4-97143771efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" Nov 03 20:15:06.172449 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.172381 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65ae418c-6xl2l\" (UID: \"acab3741-a3ca-415d-b8e4-97143771efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" Nov 03 20:15:06.273822 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.273785 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65ae418c-6xl2l\" (UID: \"acab3741-a3ca-415d-b8e4-97143771efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" Nov 03 20:15:06.274029 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.273902 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k62c\" (UniqueName: \"kubernetes.io/projected/acab3741-a3ca-415d-b8e4-97143771efb9-kube-api-access-4k62c\") pod \"maas-api-revoke-sub-65ae418c-6xl2l\" (UID: \"acab3741-a3ca-415d-b8e4-97143771efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" Nov 03 20:15:06.274029 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.274008 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65ae418c-6xl2l\" (UID: \"acab3741-a3ca-415d-b8e4-97143771efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" Nov 03 20:15:06.274635 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.274606 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65ae418c-6xl2l\" (UID: \"acab3741-a3ca-415d-b8e4-97143771efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" Nov 03 20:15:06.276739 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.276713 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65ae418c-6xl2l\" (UID: \"acab3741-a3ca-415d-b8e4-97143771efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" Nov 03 20:15:06.296333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.296307 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k62c\" (UniqueName: \"kubernetes.io/projected/acab3741-a3ca-415d-b8e4-97143771efb9-kube-api-access-4k62c\") pod \"maas-api-revoke-sub-65ae418c-6xl2l\" (UID: \"acab3741-a3ca-415d-b8e4-97143771efb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" Nov 03 20:15:06.411287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.411210 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" Nov 03 20:15:06.705791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:06.705756 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l"] Nov 03 20:15:06.706754 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:15:06.706721 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacab3741_a3ca_415d_b8e4_97143771efb9.slice/crio-11a6bf477135623a085afacaa6246fcce568b2f0531dc578691252c01c0d9f55 WatchSource:0}: Error finding container 11a6bf477135623a085afacaa6246fcce568b2f0531dc578691252c01c0d9f55: Status 404 returned error can't find the container with id 11a6bf477135623a085afacaa6246fcce568b2f0531dc578691252c01c0d9f55 Nov 03 20:15:07.180786 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:07.180746 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" event={"ID":"acab3741-a3ca-415d-b8e4-97143771efb9","Type":"ContainerStarted","Data":"db08e05e6a2b8a5ac991f850c3d46f42c65c994e252f4f0a483effd60f85b9b9"} Nov 03 20:15:07.181261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:07.180811 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" event={"ID":"acab3741-a3ca-415d-b8e4-97143771efb9","Type":"ContainerStarted","Data":"11a6bf477135623a085afacaa6246fcce568b2f0531dc578691252c01c0d9f55"} Nov 03 20:15:07.200601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:07.200533 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" podStartSLOduration=1.200513884 podStartE2EDuration="1.200513884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:07.198900165 +0000 UTC m=+1085.574496914" watchObservedRunningTime="2026-11-03 20:15:07.200513884 +0000 UTC m=+1085.576110635" Nov 03 20:15:08.191433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.191369 4457 generic.go:356] "Generic (PLEG): container finished" podID="acab3741-a3ca-415d-b8e4-97143771efb9" containerID="db08e05e6a2b8a5ac991f850c3d46f42c65c994e252f4f0a483effd60f85b9b9" exitCode=0 Nov 03 20:15:08.191840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.191459 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" event={"ID":"acab3741-a3ca-415d-b8e4-97143771efb9","Type":"ContainerDied","Data":"db08e05e6a2b8a5ac991f850c3d46f42c65c994e252f4f0a483effd60f85b9b9"} Nov 03 20:15:08.618614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.618573 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md"] Nov 03 20:15:08.623121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.623097 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" Nov 03 20:15:08.632716 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.632688 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md"] Nov 03 20:15:08.715642 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.715601 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa0a88a-pv8md\" (UID: \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" Nov 03 20:15:08.715797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.715737 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndgj\" (UniqueName: \"kubernetes.io/projected/f216dffd-5ec3-4ec9-8c40-9534f69ee329-kube-api-access-jndgj\") pod \"maas-api-revoke-sub-baa0a88a-pv8md\" (UID: \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" Nov 03 20:15:08.715983 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.715964 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa0a88a-pv8md\" (UID: \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" Nov 03 20:15:08.817964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.817926 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa0a88a-pv8md\" (UID: \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" Nov 03 20:15:08.818134 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.818080 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa0a88a-pv8md\" (UID: \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" Nov 03 20:15:08.818186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.818151 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndgj\" (UniqueName: \"kubernetes.io/projected/f216dffd-5ec3-4ec9-8c40-9534f69ee329-kube-api-access-jndgj\") pod \"maas-api-revoke-sub-baa0a88a-pv8md\" (UID: \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" Nov 03 20:15:08.818802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.818775 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baa0a88a-pv8md\" (UID: \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" Nov 03 20:15:08.820573 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.820546 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baa0a88a-pv8md\" (UID: \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" Nov 03 20:15:08.833762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.833728 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndgj\" (UniqueName: \"kubernetes.io/projected/f216dffd-5ec3-4ec9-8c40-9534f69ee329-kube-api-access-jndgj\") pod \"maas-api-revoke-sub-baa0a88a-pv8md\" (UID: \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" Nov 03 20:15:08.965186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:08.965093 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" Nov 03 20:15:09.299030 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:15:09.298985 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf216dffd_5ec3_4ec9_8c40_9534f69ee329.slice/crio-36e1c6a6eef59767ebe03bd68e2cbfc3c9892df6dc813195d8f44aeb195fd849 WatchSource:0}: Error finding container 36e1c6a6eef59767ebe03bd68e2cbfc3c9892df6dc813195d8f44aeb195fd849: Status 404 returned error can't find the container with id 36e1c6a6eef59767ebe03bd68e2cbfc3c9892df6dc813195d8f44aeb195fd849 Nov 03 20:15:09.308505 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:09.301620 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md"] Nov 03 20:15:09.433678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:09.433652 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" Nov 03 20:15:09.533427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:09.533306 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-xks-ca\") pod \"acab3741-a3ca-415d-b8e4-97143771efb9\" (UID: \"acab3741-a3ca-415d-b8e4-97143771efb9\") " Nov 03 20:15:09.533427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:09.533403 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-openshift-ca\") pod \"acab3741-a3ca-415d-b8e4-97143771efb9\" (UID: \"acab3741-a3ca-415d-b8e4-97143771efb9\") " Nov 03 20:15:09.533650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:09.533610 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acab3741-a3ca-415d-b8e4-97143771efb9-kube-api-access-4k62c\" (UniqueName: \"kubernetes.io/projected/acab3741-a3ca-415d-b8e4-97143771efb9-kube-api-access-4k62c\") pod \"acab3741-a3ca-415d-b8e4-97143771efb9\" (UID: \"acab3741-a3ca-415d-b8e4-97143771efb9\") " Nov 03 20:15:09.534081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:09.534047 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-openshift-ca" pod "acab3741-a3ca-415d-b8e4-97143771efb9" (UID: "acab3741-a3ca-415d-b8e4-97143771efb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:09.534436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:09.534414 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:09.558037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:09.557992 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-xks-ca" pod "acab3741-a3ca-415d-b8e4-97143771efb9" (UID: "acab3741-a3ca-415d-b8e4-97143771efb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:09.558525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:09.558497 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acab3741-a3ca-415d-b8e4-97143771efb9-kube-api-access-4k62c" pod "acab3741-a3ca-415d-b8e4-97143771efb9" (UID: "acab3741-a3ca-415d-b8e4-97143771efb9"). InnerVolumeSpecName "kube-api-access-4k62c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:09.636095 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:09.636061 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k62c\" (UniqueName: \"kubernetes.io/projected/acab3741-a3ca-415d-b8e4-97143771efb9-kube-api-access-4k62c\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:09.636095 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:09.636092 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acab3741-a3ca-415d-b8e4-97143771efb9-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:10.215555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:10.215473 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" Nov 03 20:15:10.215555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:10.215504 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l" event={"ID":"acab3741-a3ca-415d-b8e4-97143771efb9","Type":"ContainerDied","Data":"11a6bf477135623a085afacaa6246fcce568b2f0531dc578691252c01c0d9f55"} Nov 03 20:15:10.215555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:10.215555 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11a6bf477135623a085afacaa6246fcce568b2f0531dc578691252c01c0d9f55" Nov 03 20:15:10.217477 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:10.217442 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" event={"ID":"f216dffd-5ec3-4ec9-8c40-9534f69ee329","Type":"ContainerStarted","Data":"dd649d122aee7dac604de6ffe3bef55fde84796158ae46964aa811fc0d2fc078"} Nov 03 20:15:10.217616 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:10.217503 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" event={"ID":"f216dffd-5ec3-4ec9-8c40-9534f69ee329","Type":"ContainerStarted","Data":"36e1c6a6eef59767ebe03bd68e2cbfc3c9892df6dc813195d8f44aeb195fd849"} Nov 03 20:15:10.237349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:10.237303 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" podStartSLOduration=2.237289853 podStartE2EDuration="2.237289853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:10.236653453 +0000 UTC m=+1088.612250202" watchObservedRunningTime="2026-11-03 20:15:10.237289853 +0000 UTC m=+1088.612886638" Nov 03 20:15:11.231539 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:11.231500 4457 generic.go:356] "Generic (PLEG): container finished" podID="f216dffd-5ec3-4ec9-8c40-9534f69ee329" containerID="dd649d122aee7dac604de6ffe3bef55fde84796158ae46964aa811fc0d2fc078" exitCode=0 Nov 03 20:15:11.232065 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:11.231582 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" event={"ID":"f216dffd-5ec3-4ec9-8c40-9534f69ee329","Type":"ContainerDied","Data":"dd649d122aee7dac604de6ffe3bef55fde84796158ae46964aa811fc0d2fc078"} Nov 03 20:15:12.431818 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:12.431788 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" Nov 03 20:15:12.593104 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:12.593070 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f216dffd-5ec3-4ec9-8c40-9534f69ee329-kube-api-access-jndgj\" (UniqueName: \"kubernetes.io/projected/f216dffd-5ec3-4ec9-8c40-9534f69ee329-kube-api-access-jndgj\") pod \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\" (UID: \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\") " Nov 03 20:15:12.593466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:12.593435 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-xks-ca\") pod \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\" (UID: \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\") " Nov 03 20:15:12.593679 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:12.593653 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-openshift-ca\") pod \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\" (UID: \"f216dffd-5ec3-4ec9-8c40-9534f69ee329\") " Nov 03 20:15:12.594882 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:12.594853 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-openshift-ca" pod "f216dffd-5ec3-4ec9-8c40-9534f69ee329" (UID: "f216dffd-5ec3-4ec9-8c40-9534f69ee329"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:12.596441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:12.595811 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f216dffd-5ec3-4ec9-8c40-9534f69ee329-kube-api-access-jndgj" pod "f216dffd-5ec3-4ec9-8c40-9534f69ee329" (UID: "f216dffd-5ec3-4ec9-8c40-9534f69ee329"). InnerVolumeSpecName "kube-api-access-jndgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:12.600178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:12.600153 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-xks-ca" pod "f216dffd-5ec3-4ec9-8c40-9534f69ee329" (UID: "f216dffd-5ec3-4ec9-8c40-9534f69ee329"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:12.696767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:12.696720 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:12.696767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:12.696764 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f216dffd-5ec3-4ec9-8c40-9534f69ee329-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:12.696767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:12.696780 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jndgj\" (UniqueName: \"kubernetes.io/projected/f216dffd-5ec3-4ec9-8c40-9534f69ee329-kube-api-access-jndgj\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:13.249833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:13.249791 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" event={"ID":"f216dffd-5ec3-4ec9-8c40-9534f69ee329","Type":"ContainerDied","Data":"36e1c6a6eef59767ebe03bd68e2cbfc3c9892df6dc813195d8f44aeb195fd849"} Nov 03 20:15:13.249833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:13.249823 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md" Nov 03 20:15:13.250038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:13.249893 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36e1c6a6eef59767ebe03bd68e2cbfc3c9892df6dc813195d8f44aeb195fd849" Nov 03 20:15:14.011969 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.011930 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk"] Nov 03 20:15:14.012863 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.012840 4457 cpu_manager.go:414] "containerMap: removing container" podUID="f216dffd-5ec3-4ec9-8c40-9534f69ee329" containerName="revoke-keys" Nov 03 20:15:14.012863 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.012863 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f216dffd-5ec3-4ec9-8c40-9534f69ee329" containerName="revoke-keys" Nov 03 20:15:14.012987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.012875 4457 cpu_manager.go:414] "containerMap: removing container" podUID="acab3741-a3ca-415d-b8e4-97143771efb9" containerName="revoke-keys" Nov 03 20:15:14.012987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.012880 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acab3741-a3ca-415d-b8e4-97143771efb9" containerName="revoke-keys" Nov 03 20:15:14.013107 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.013009 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="f216dffd-5ec3-4ec9-8c40-9534f69ee329" containerName="revoke-keys" Nov 03 20:15:14.013107 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.013019 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="acab3741-a3ca-415d-b8e4-97143771efb9" containerName="revoke-keys" Nov 03 20:15:14.016911 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.016885 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" Nov 03 20:15:14.019574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.019547 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f87c167e-sj4bk\" (UID: \"00169716-105a-4f81-a435-bf13cfb20a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" Nov 03 20:15:14.019792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.019761 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f87c167e-sj4bk\" (UID: \"00169716-105a-4f81-a435-bf13cfb20a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" Nov 03 20:15:14.019902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.019851 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ss7s\" (UniqueName: \"kubernetes.io/projected/00169716-105a-4f81-a435-bf13cfb20a8d-kube-api-access-8ss7s\") pod \"maas-api-revoke-sub-f87c167e-sj4bk\" (UID: \"00169716-105a-4f81-a435-bf13cfb20a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" Nov 03 20:15:14.024606 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.024565 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:14.026257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.026232 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk"] Nov 03 20:15:14.120907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.120869 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ss7s\" (UniqueName: \"kubernetes.io/projected/00169716-105a-4f81-a435-bf13cfb20a8d-kube-api-access-8ss7s\") pod \"maas-api-revoke-sub-f87c167e-sj4bk\" (UID: \"00169716-105a-4f81-a435-bf13cfb20a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" Nov 03 20:15:14.121139 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.121112 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f87c167e-sj4bk\" (UID: \"00169716-105a-4f81-a435-bf13cfb20a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" Nov 03 20:15:14.121222 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.121205 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f87c167e-sj4bk\" (UID: \"00169716-105a-4f81-a435-bf13cfb20a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" Nov 03 20:15:14.121993 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.121964 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f87c167e-sj4bk\" (UID: \"00169716-105a-4f81-a435-bf13cfb20a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" Nov 03 20:15:14.124351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.124327 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f87c167e-sj4bk\" (UID: \"00169716-105a-4f81-a435-bf13cfb20a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" Nov 03 20:15:14.133184 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.133159 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ss7s\" (UniqueName: \"kubernetes.io/projected/00169716-105a-4f81-a435-bf13cfb20a8d-kube-api-access-8ss7s\") pod \"maas-api-revoke-sub-f87c167e-sj4bk\" (UID: \"00169716-105a-4f81-a435-bf13cfb20a8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" Nov 03 20:15:14.347904 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.347867 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" Nov 03 20:15:14.648464 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:15:14.648430 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00169716_105a_4f81_a435_bf13cfb20a8d.slice/crio-4ecf3b5071fd4ee5b568788efe369deb24479a61360e6235451fa0f7f7fd3924 WatchSource:0}: Error finding container 4ecf3b5071fd4ee5b568788efe369deb24479a61360e6235451fa0f7f7fd3924: Status 404 returned error can't find the container with id 4ecf3b5071fd4ee5b568788efe369deb24479a61360e6235451fa0f7f7fd3924 Nov 03 20:15:14.649696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.649668 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk"] Nov 03 20:15:14.750302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:14.750271 4457 scope.go:122] "RemoveContainer" containerID="a0cb21909def09780587d124764de283b0cdfa22ccb5853a5ac0284259150964" Nov 03 20:15:15.276817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:15.276716 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" event={"ID":"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c","Type":"ContainerStarted","Data":"47555fcec53f5e92aa2785c36597bee51bae0b74bd8913892411d9456305955d"} Nov 03 20:15:15.278892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:15.278863 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" event={"ID":"00169716-105a-4f81-a435-bf13cfb20a8d","Type":"ContainerStarted","Data":"496cf619788d0b0d33498a12d17650336d944e276550e3585b458e2964d26e35"} Nov 03 20:15:15.279026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:15.278916 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" event={"ID":"00169716-105a-4f81-a435-bf13cfb20a8d","Type":"ContainerStarted","Data":"4ecf3b5071fd4ee5b568788efe369deb24479a61360e6235451fa0f7f7fd3924"} Nov 03 20:15:15.320626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:15.320566 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" podStartSLOduration=2.320549919 podStartE2EDuration="2.320549919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:15.317353922 +0000 UTC m=+1093.692950670" watchObservedRunningTime="2026-11-03 20:15:15.320549919 +0000 UTC m=+1093.696146666" Nov 03 20:15:15.749985 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:15.749955 4457 scope.go:122] "RemoveContainer" containerID="fd2656867f7475afb332b3b88da09fbda88752f359491b2e37640fcb48a9d3f0" Nov 03 20:15:16.309468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.309432 4457 generic.go:356] "Generic (PLEG): container finished" podID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" containerID="47555fcec53f5e92aa2785c36597bee51bae0b74bd8913892411d9456305955d" exitCode=60 Nov 03 20:15:16.309965 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.309525 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" event={"ID":"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c","Type":"ContainerDied","Data":"47555fcec53f5e92aa2785c36597bee51bae0b74bd8913892411d9456305955d"} Nov 03 20:15:16.310047 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.309973 4457 scope.go:122] "RemoveContainer" containerID="a0cb21909def09780587d124764de283b0cdfa22ccb5853a5ac0284259150964" Nov 03 20:15:16.310433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.310402 4457 scope.go:122] "RemoveContainer" containerID="47555fcec53f5e92aa2785c36597bee51bae0b74bd8913892411d9456305955d" Nov 03 20:15:16.310819 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:16.310782 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9_odh-ai-gateway-infra(dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" podUID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" Nov 03 20:15:16.311889 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.311867 4457 generic.go:356] "Generic (PLEG): container finished" podID="00169716-105a-4f81-a435-bf13cfb20a8d" containerID="496cf619788d0b0d33498a12d17650336d944e276550e3585b458e2964d26e35" exitCode=0 Nov 03 20:15:16.311976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.311943 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" event={"ID":"00169716-105a-4f81-a435-bf13cfb20a8d","Type":"ContainerDied","Data":"496cf619788d0b0d33498a12d17650336d944e276550e3585b458e2964d26e35"} Nov 03 20:15:16.314696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.314672 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" event={"ID":"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a","Type":"ContainerStarted","Data":"6570632e6c732629426af593e6f3e19b7278c9e337bd680f48dcb372904f6847"} Nov 03 20:15:16.325442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.325419 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9"] Nov 03 20:15:16.349629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.349601 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk"] Nov 03 20:15:16.360418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.356486 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" Nov 03 20:15:16.388478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.388446 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk"] Nov 03 20:15:16.436352 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.436322 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26"] Nov 03 20:15:16.440949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.440926 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.462626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.461628 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.462626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.461676 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5rwg\" (UniqueName: \"kubernetes.io/projected/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-kube-api-access-z5rwg\") pod \"maas-api-revoke-sub-6c89d401-xrmkk\" (UID: \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" Nov 03 20:15:16.462626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.461703 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.462626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.461757 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b5e918-acce-4fd9-ad4a-c37d87965667-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.462626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.461789 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.462626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.461858 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.462626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.461920 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.462626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.461975 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.462626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.461993 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp4zs\" (UniqueName: \"kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-kube-api-access-vp4zs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.462626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.462055 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c89d401-xrmkk\" (UID: \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" Nov 03 20:15:16.462626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.462085 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c89d401-xrmkk\" (UID: \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" Nov 03 20:15:16.462626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.462099 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.465650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.465624 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26"] Nov 03 20:15:16.519116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.519075 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp"] Nov 03 20:15:16.523641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.523619 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" Nov 03 20:15:16.539439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.539413 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp"] Nov 03 20:15:16.568248 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568152 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c89d401-xrmkk\" (UID: \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" Nov 03 20:15:16.568248 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568201 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c89d401-xrmkk\" (UID: \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" Nov 03 20:15:16.568248 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568219 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.568248 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568242 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bgbk\" (UniqueName: \"kubernetes.io/projected/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-kube-api-access-5bgbk\") pod \"maas-api-revoke-sub-7069a6d9-p2slp\" (UID: \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" Nov 03 20:15:16.568582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568277 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.568582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568313 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5rwg\" (UniqueName: \"kubernetes.io/projected/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-kube-api-access-z5rwg\") pod \"maas-api-revoke-sub-6c89d401-xrmkk\" (UID: \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" Nov 03 20:15:16.568582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568340 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.568582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568398 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b5e918-acce-4fd9-ad4a-c37d87965667-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.568582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568420 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7069a6d9-p2slp\" (UID: \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" Nov 03 20:15:16.568582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568446 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.568582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568523 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.568582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568576 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.568981 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568631 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.568981 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568647 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp4zs\" (UniqueName: \"kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-kube-api-access-vp4zs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.568981 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.568691 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7069a6d9-p2slp\" (UID: \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" Nov 03 20:15:16.569589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.569433 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c89d401-xrmkk\" (UID: \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" Nov 03 20:15:16.570739 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.570712 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-data\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.570898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.570749 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-certs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.571009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.570822 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-credential-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.571188 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.571164 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b5e918-acce-4fd9-ad4a-c37d87965667-istiod-ca-cert\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.571404 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.571357 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-socket\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.573868 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.573806 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-podinfo\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.573987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.573889 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c89d401-xrmkk\" (UID: \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" Nov 03 20:15:16.575678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.575634 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-envoy\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.592576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.592513 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp4zs\" (UniqueName: \"kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-kube-api-access-vp4zs\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.600076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.599942 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5rwg\" (UniqueName: \"kubernetes.io/projected/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-kube-api-access-z5rwg\") pod \"maas-api-revoke-sub-6c89d401-xrmkk\" (UID: \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" Nov 03 20:15:16.600565 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.600525 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-token\") pod \"e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.678283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.671520 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7069a6d9-p2slp\" (UID: \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" Nov 03 20:15:16.678283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.674546 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bgbk\" (UniqueName: \"kubernetes.io/projected/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-kube-api-access-5bgbk\") pod \"maas-api-revoke-sub-7069a6d9-p2slp\" (UID: \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" Nov 03 20:15:16.678283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.674833 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7069a6d9-p2slp\" (UID: \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" Nov 03 20:15:16.678283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.676857 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7069a6d9-p2slp\" (UID: \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" Nov 03 20:15:16.679514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.679490 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" Nov 03 20:15:16.682213 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.681419 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7069a6d9-p2slp\" (UID: \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" Nov 03 20:15:16.714256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.714214 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bgbk\" (UniqueName: \"kubernetes.io/projected/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-kube-api-access-5bgbk\") pod \"maas-api-revoke-sub-7069a6d9-p2slp\" (UID: \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" Nov 03 20:15:16.750728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.750695 4457 scope.go:122] "RemoveContainer" containerID="b9f269b8052724c32ec59cceda48bc4c3d556d05e22c29a8635642449ba40f84" Nov 03 20:15:16.755230 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.754989 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:16.842250 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:16.842216 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" Nov 03 20:15:17.013259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.013211 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk"] Nov 03 20:15:17.014059 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:15:17.014028 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fa0c0d_8a71_4ffa_ac05_04653af8b63c.slice/crio-64e63ad47a0d32139fcdca0219341099eee6742c30b4143d568908dcebca0577 WatchSource:0}: Error finding container 64e63ad47a0d32139fcdca0219341099eee6742c30b4143d568908dcebca0577: Status 404 returned error can't find the container with id 64e63ad47a0d32139fcdca0219341099eee6742c30b4143d568908dcebca0577 Nov 03 20:15:17.114231 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.114176 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:17.114841 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.114336 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:17.124905 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.118829 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:17.140183 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.140147 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26"] Nov 03 20:15:17.191609 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.191586 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp"] Nov 03 20:15:17.262440 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:15:17.262370 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81ab721_84a4_4df3_bcde_91a7b2c7cc8d.slice/crio-31d604ffd1c69ab3cdfcb3e8817de4014b2e0033fb8774c05127bb483bc2e1e4 WatchSource:0}: Error finding container 31d604ffd1c69ab3cdfcb3e8817de4014b2e0033fb8774c05127bb483bc2e1e4: Status 404 returned error can't find the container with id 31d604ffd1c69ab3cdfcb3e8817de4014b2e0033fb8774c05127bb483bc2e1e4 Nov 03 20:15:17.327142 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.327106 4457 generic.go:356] "Generic (PLEG): container finished" podID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" containerID="6570632e6c732629426af593e6f3e19b7278c9e337bd680f48dcb372904f6847" exitCode=60 Nov 03 20:15:17.327636 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.327178 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" event={"ID":"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a","Type":"ContainerDied","Data":"6570632e6c732629426af593e6f3e19b7278c9e337bd680f48dcb372904f6847"} Nov 03 20:15:17.327636 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.327244 4457 scope.go:122] "RemoveContainer" containerID="fd2656867f7475afb332b3b88da09fbda88752f359491b2e37640fcb48a9d3f0" Nov 03 20:15:17.327762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.327740 4457 scope.go:122] "RemoveContainer" containerID="6570632e6c732629426af593e6f3e19b7278c9e337bd680f48dcb372904f6847" Nov 03 20:15:17.328218 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:17.328184 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg_odh-ai-gateway-infra(809babf5-bc1f-4a26-8fbd-67f1a5d0f11a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" podUID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" Nov 03 20:15:17.329430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.329370 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" event={"ID":"7cc9e13f-05f6-4a9e-a15c-034c176b4695","Type":"ContainerStarted","Data":"033b2b35cdeaff7271d901df31de22e2eb7d2d400d63f9ff44721c986f4ad624"} Nov 03 20:15:17.330761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.330730 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" event={"ID":"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c","Type":"ContainerStarted","Data":"64e63ad47a0d32139fcdca0219341099eee6742c30b4143d568908dcebca0577"} Nov 03 20:15:17.332745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.332716 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" event={"ID":"94b5e918-acce-4fd9-ad4a-c37d87965667","Type":"ContainerStarted","Data":"7c4d688bf3e753541e2e7efe52d0579b9c3086d36814b5657b4f4339b86184df"} Nov 03 20:15:17.332844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.332772 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" event={"ID":"94b5e918-acce-4fd9-ad4a-c37d87965667","Type":"ContainerStarted","Data":"a6e9fc6435397fe44d886fffe9ddba733a4369044f267e04f1e40afbc124c316"} Nov 03 20:15:17.337172 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.337150 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" event={"ID":"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d","Type":"ContainerStarted","Data":"31d604ffd1c69ab3cdfcb3e8817de4014b2e0033fb8774c05127bb483bc2e1e4"} Nov 03 20:15:17.376695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.376573 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" podStartSLOduration=1.376553734 podStartE2EDuration="1.376553734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:17.372658546 +0000 UTC m=+1095.748255294" watchObservedRunningTime="2026-11-03 20:15:17.376553734 +0000 UTC m=+1095.752150482" Nov 03 20:15:17.405827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.395796 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg"] Nov 03 20:15:17.650487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.650401 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" Nov 03 20:15:17.705246 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.705218 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" Nov 03 20:15:17.727879 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.727841 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-kube-api-access-hfprv\" (UniqueName: \"kubernetes.io/projected/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-kube-api-access-hfprv\") pod \"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c\" (UID: \"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c\") " Nov 03 20:15:17.728140 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.728116 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-maas-api-service-ca\") pod \"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c\" (UID: \"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c\") " Nov 03 20:15:17.729377 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.729347 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-maas-api-service-ca" pod "dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" (UID: "dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:17.736989 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.736955 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-kube-api-access-hfprv" pod "dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" (UID: "dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c"). InnerVolumeSpecName "kube-api-access-hfprv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:17.817483 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.817459 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:17.845665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.841037 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-openshift-ca\") pod \"00169716-105a-4f81-a435-bf13cfb20a8d\" (UID: \"00169716-105a-4f81-a435-bf13cfb20a8d\") " Nov 03 20:15:17.845665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.841123 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00169716-105a-4f81-a435-bf13cfb20a8d-kube-api-access-8ss7s\" (UniqueName: \"kubernetes.io/projected/00169716-105a-4f81-a435-bf13cfb20a8d-kube-api-access-8ss7s\") pod \"00169716-105a-4f81-a435-bf13cfb20a8d\" (UID: \"00169716-105a-4f81-a435-bf13cfb20a8d\") " Nov 03 20:15:17.845665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.841374 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-xks-ca\") pod \"00169716-105a-4f81-a435-bf13cfb20a8d\" (UID: \"00169716-105a-4f81-a435-bf13cfb20a8d\") " Nov 03 20:15:17.845665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.842379 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:17.845665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.842418 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfprv\" (UniqueName: \"kubernetes.io/projected/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c-kube-api-access-hfprv\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:17.845665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.842431 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-openshift-ca" pod "00169716-105a-4f81-a435-bf13cfb20a8d" (UID: "00169716-105a-4f81-a435-bf13cfb20a8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:17.847919 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.846844 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-xks-ca" pod "00169716-105a-4f81-a435-bf13cfb20a8d" (UID: "00169716-105a-4f81-a435-bf13cfb20a8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:17.863622 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.863551 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00169716-105a-4f81-a435-bf13cfb20a8d-kube-api-access-8ss7s" pod "00169716-105a-4f81-a435-bf13cfb20a8d" (UID: "00169716-105a-4f81-a435-bf13cfb20a8d"). InnerVolumeSpecName "kube-api-access-8ss7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:17.944428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.944335 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:17.944428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.944362 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00169716-105a-4f81-a435-bf13cfb20a8d-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:17.944428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:17.944372 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ss7s\" (UniqueName: \"kubernetes.io/projected/00169716-105a-4f81-a435-bf13cfb20a8d-kube-api-access-8ss7s\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:18.412564 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.412523 4457 generic.go:356] "Generic (PLEG): container finished" podID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" containerID="033b2b35cdeaff7271d901df31de22e2eb7d2d400d63f9ff44721c986f4ad624" exitCode=60 Nov 03 20:15:18.413019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.412694 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" event={"ID":"7cc9e13f-05f6-4a9e-a15c-034c176b4695","Type":"ContainerDied","Data":"033b2b35cdeaff7271d901df31de22e2eb7d2d400d63f9ff44721c986f4ad624"} Nov 03 20:15:18.413019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.412750 4457 scope.go:122] "RemoveContainer" containerID="b9f269b8052724c32ec59cceda48bc4c3d556d05e22c29a8635642449ba40f84" Nov 03 20:15:18.414220 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.414195 4457 scope.go:122] "RemoveContainer" containerID="033b2b35cdeaff7271d901df31de22e2eb7d2d400d63f9ff44721c986f4ad624" Nov 03 20:15:18.416113 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:18.416068 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9_odh-ai-gateway-infra(7cc9e13f-05f6-4a9e-a15c-034c176b4695)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" podUID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" Nov 03 20:15:18.418913 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.418848 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" event={"ID":"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c","Type":"ContainerStarted","Data":"98dea328c5ec30894c9ab86c9e326323c159a22ceaa322b17248970afafc8441"} Nov 03 20:15:18.423454 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.422970 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" event={"ID":"dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c","Type":"ContainerDied","Data":"0086cc6fb1edc1e843d8503edc319af25cc509d5967139182c741d33dac32a85"} Nov 03 20:15:18.423454 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.423082 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9" Nov 03 20:15:18.431097 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.431069 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" Nov 03 20:15:18.432144 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.432116 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk" event={"ID":"00169716-105a-4f81-a435-bf13cfb20a8d","Type":"ContainerDied","Data":"4ecf3b5071fd4ee5b568788efe369deb24479a61360e6235451fa0f7f7fd3924"} Nov 03 20:15:18.432260 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.432170 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ecf3b5071fd4ee5b568788efe369deb24479a61360e6235451fa0f7f7fd3924" Nov 03 20:15:18.439213 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.439006 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9"] Nov 03 20:15:18.440857 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.440828 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" event={"ID":"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d","Type":"ContainerStarted","Data":"e9486c42f7d40554102c62042b9a826268c5fe9d9df0142f2e6c9d12e0292616"} Nov 03 20:15:18.533006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.532631 4457 scope.go:122] "RemoveContainer" containerID="47555fcec53f5e92aa2785c36597bee51bae0b74bd8913892411d9456305955d" Nov 03 20:15:18.541912 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.541855 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" podStartSLOduration=2.541836357 podStartE2EDuration="2.541836357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:18.520238321 +0000 UTC m=+1096.895835068" watchObservedRunningTime="2026-11-03 20:15:18.541836357 +0000 UTC m=+1096.917433106" Nov 03 20:15:18.613237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.613083 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9"] Nov 03 20:15:18.625702 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.625169 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895615-5swd9"] Nov 03 20:15:18.809164 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.809121 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 20:15:18.809369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.809214 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" podUID="94b5e918-acce-4fd9-ad4a-c37d87965667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 20:15:18.836498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.836049 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" Nov 03 20:15:18.938796 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.938712 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-kube-api-access-mfxnd\" (UniqueName: \"kubernetes.io/projected/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-kube-api-access-mfxnd\") pod \"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a\" (UID: \"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a\") " Nov 03 20:15:18.939146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.939120 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-maas-api-service-ca\") pod \"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a\" (UID: \"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a\") " Nov 03 20:15:18.939576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.939551 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-maas-api-service-ca" pod "809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" (UID: "809babf5-bc1f-4a26-8fbd-67f1a5d0f11a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:18.939957 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.939928 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:18.940697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:18.940673 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-kube-api-access-mfxnd" pod "809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" (UID: "809babf5-bc1f-4a26-8fbd-67f1a5d0f11a"). InnerVolumeSpecName "kube-api-access-mfxnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:19.042055 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.042015 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfxnd\" (UniqueName: \"kubernetes.io/projected/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a-kube-api-access-mfxnd\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:19.450466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.450422 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" event={"ID":"809babf5-bc1f-4a26-8fbd-67f1a5d0f11a","Type":"ContainerDied","Data":"4a74ace746908bf20ae9dcf75b6b7a5403da5cbb960378123bc364a0f18c1581"} Nov 03 20:15:19.450466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.450456 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg" Nov 03 20:15:19.451075 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.450505 4457 scope.go:122] "RemoveContainer" containerID="6570632e6c732629426af593e6f3e19b7278c9e337bd680f48dcb372904f6847" Nov 03 20:15:19.454479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.454441 4457 generic.go:356] "Generic (PLEG): container finished" podID="c6fa0c0d-8a71-4ffa-ac05-04653af8b63c" containerID="98dea328c5ec30894c9ab86c9e326323c159a22ceaa322b17248970afafc8441" exitCode=0 Nov 03 20:15:19.454598 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.454520 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" event={"ID":"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c","Type":"ContainerDied","Data":"98dea328c5ec30894c9ab86c9e326323c159a22ceaa322b17248970afafc8441"} Nov 03 20:15:19.504949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.504454 4457 generic.go:356] "Generic (PLEG): container finished" podID="e81ab721-84a4-4df3-bcde-91a7b2c7cc8d" containerID="e9486c42f7d40554102c62042b9a826268c5fe9d9df0142f2e6c9d12e0292616" exitCode=0 Nov 03 20:15:19.506837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.505929 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" event={"ID":"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d","Type":"ContainerDied","Data":"e9486c42f7d40554102c62042b9a826268c5fe9d9df0142f2e6c9d12e0292616"} Nov 03 20:15:19.546787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.546741 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg"] Nov 03 20:15:19.564683 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.554770 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895615-j9lsg"] Nov 03 20:15:19.832714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.830799 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 20:15:19.832714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.830860 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" podUID="94b5e918-acce-4fd9-ad4a-c37d87965667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 20:15:19.901683 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.897243 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" path="/var/lib/kubelet/pods/809babf5-bc1f-4a26-8fbd-67f1a5d0f11a/volumes" Nov 03 20:15:19.901683 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.897932 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" path="/var/lib/kubelet/pods/dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c/volumes" Nov 03 20:15:19.903210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.903184 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" Nov 03 20:15:19.940104 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:19.939832 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" Nov 03 20:15:20.030722 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.030632 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cc9e13f-05f6-4a9e-a15c-034c176b4695-kube-api-access-bmfzb\" (UniqueName: \"kubernetes.io/projected/7cc9e13f-05f6-4a9e-a15c-034c176b4695-kube-api-access-bmfzb\") pod \"7cc9e13f-05f6-4a9e-a15c-034c176b4695\" (UID: \"7cc9e13f-05f6-4a9e-a15c-034c176b4695\") " Nov 03 20:15:20.030722 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.030686 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cc9e13f-05f6-4a9e-a15c-034c176b4695-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc9e13f-05f6-4a9e-a15c-034c176b4695-maas-api-service-ca\") pod \"7cc9e13f-05f6-4a9e-a15c-034c176b4695\" (UID: \"7cc9e13f-05f6-4a9e-a15c-034c176b4695\") " Nov 03 20:15:20.031701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.031672 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc9e13f-05f6-4a9e-a15c-034c176b4695-maas-api-service-ca" pod "7cc9e13f-05f6-4a9e-a15c-034c176b4695" (UID: "7cc9e13f-05f6-4a9e-a15c-034c176b4695"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:20.032897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.032873 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc9e13f-05f6-4a9e-a15c-034c176b4695-kube-api-access-bmfzb" pod "7cc9e13f-05f6-4a9e-a15c-034c176b4695" (UID: "7cc9e13f-05f6-4a9e-a15c-034c176b4695"). InnerVolumeSpecName "kube-api-access-bmfzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:20.132462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.132432 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-openshift-ca\") pod \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\" (UID: \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\") " Nov 03 20:15:20.132630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.132570 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-xks-ca\") pod \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\" (UID: \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\") " Nov 03 20:15:20.132697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.132667 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-kube-api-access-5bgbk\" (UniqueName: \"kubernetes.io/projected/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-kube-api-access-5bgbk\") pod \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\" (UID: \"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d\") " Nov 03 20:15:20.132889 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.132854 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-openshift-ca" pod "e81ab721-84a4-4df3-bcde-91a7b2c7cc8d" (UID: "e81ab721-84a4-4df3-bcde-91a7b2c7cc8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:20.133255 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.133231 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmfzb\" (UniqueName: \"kubernetes.io/projected/7cc9e13f-05f6-4a9e-a15c-034c176b4695-kube-api-access-bmfzb\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.133371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.133259 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc9e13f-05f6-4a9e-a15c-034c176b4695-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.133371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.133273 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.134781 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.134754 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-xks-ca" pod "e81ab721-84a4-4df3-bcde-91a7b2c7cc8d" (UID: "e81ab721-84a4-4df3-bcde-91a7b2c7cc8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:20.134862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.134841 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-kube-api-access-5bgbk" pod "e81ab721-84a4-4df3-bcde-91a7b2c7cc8d" (UID: "e81ab721-84a4-4df3-bcde-91a7b2c7cc8d"). InnerVolumeSpecName "kube-api-access-5bgbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:20.234464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.234426 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.234464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.234455 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bgbk\" (UniqueName: \"kubernetes.io/projected/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d-kube-api-access-5bgbk\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.518945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.518906 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" event={"ID":"e81ab721-84a4-4df3-bcde-91a7b2c7cc8d","Type":"ContainerDied","Data":"31d604ffd1c69ab3cdfcb3e8817de4014b2e0033fb8774c05127bb483bc2e1e4"} Nov 03 20:15:20.519471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.519333 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31d604ffd1c69ab3cdfcb3e8817de4014b2e0033fb8774c05127bb483bc2e1e4" Nov 03 20:15:20.519623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.519594 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp" Nov 03 20:15:20.522553 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.522526 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" Nov 03 20:15:20.523475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.522530 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9" event={"ID":"7cc9e13f-05f6-4a9e-a15c-034c176b4695","Type":"ContainerDied","Data":"67ffc5f97ace287f02ad1e272033ba03babbb5f0b6c7825f6218ade3163ec6f6"} Nov 03 20:15:20.523549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.523530 4457 scope.go:122] "RemoveContainer" containerID="033b2b35cdeaff7271d901df31de22e2eb7d2d400d63f9ff44721c986f4ad624" Nov 03 20:15:20.572418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.566468 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9"] Nov 03 20:15:20.584464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.581781 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-b1b190-29895615-76ql9"] Nov 03 20:15:20.743754 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.743729 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" Nov 03 20:15:20.749283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.746711 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-openshift-ca\") pod \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\" (UID: \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\") " Nov 03 20:15:20.749283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.746919 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-xks-ca\") pod \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\" (UID: \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\") " Nov 03 20:15:20.749283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.747105 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-kube-api-access-z5rwg\" (UniqueName: \"kubernetes.io/projected/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-kube-api-access-z5rwg\") pod \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\" (UID: \"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c\") " Nov 03 20:15:20.749283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.747361 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-openshift-ca" pod "c6fa0c0d-8a71-4ffa-ac05-04653af8b63c" (UID: "c6fa0c0d-8a71-4ffa-ac05-04653af8b63c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:20.749283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.747857 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.752512 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.752192 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-kube-api-access-z5rwg" pod "c6fa0c0d-8a71-4ffa-ac05-04653af8b63c" (UID: "c6fa0c0d-8a71-4ffa-ac05-04653af8b63c"). InnerVolumeSpecName "kube-api-access-z5rwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:20.875424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.872560 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-xks-ca" pod "c6fa0c0d-8a71-4ffa-ac05-04653af8b63c" (UID: "c6fa0c0d-8a71-4ffa-ac05-04653af8b63c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:20.912412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.911737 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 20:15:20.912412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.911813 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" podUID="94b5e918-acce-4fd9-ad4a-c37d87965667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 20:15:20.937201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.919413 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5rwg\" (UniqueName: \"kubernetes.io/projected/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-kube-api-access-z5rwg\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:20.937201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:20.919447 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:21.533816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:21.533789 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" Nov 03 20:15:21.534271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:21.534241 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk" event={"ID":"c6fa0c0d-8a71-4ffa-ac05-04653af8b63c","Type":"ContainerDied","Data":"64e63ad47a0d32139fcdca0219341099eee6742c30b4143d568908dcebca0577"} Nov 03 20:15:21.534457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:21.534438 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64e63ad47a0d32139fcdca0219341099eee6742c30b4143d568908dcebca0577" Nov 03 20:15:21.818628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:21.818159 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 20:15:21.818628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:21.818247 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" podUID="94b5e918-acce-4fd9-ad4a-c37d87965667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 20:15:21.851175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:21.851142 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" path="/var/lib/kubelet/pods/7cc9e13f-05f6-4a9e-a15c-034c176b4695/volumes" Nov 03 20:15:22.776413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:22.769480 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 20:15:22.776413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:22.769561 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" podUID="94b5e918-acce-4fd9-ad4a-c37d87965667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 20:15:23.757972 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:23.757938 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 20:15:23.758193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:23.758003 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" podUID="94b5e918-acce-4fd9-ad4a-c37d87965667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 20:15:24.843907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.796531 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body= Nov 03 20:15:24.846715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.846675 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" podUID="94b5e918-acce-4fd9-ad4a-c37d87965667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" Nov 03 20:15:24.847364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.830155 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7084-6ab122"] Nov 03 20:15:24.849133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.849106 4457 cpu_manager.go:414] "containerMap: removing container" podUID="e81ab721-84a4-4df3-bcde-91a7b2c7cc8d" containerName="revoke-keys" Nov 03 20:15:24.849252 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.849238 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81ab721-84a4-4df3-bcde-91a7b2c7cc8d" containerName="revoke-keys" Nov 03 20:15:24.849439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.849423 4457 cpu_manager.go:414] "containerMap: removing container" podUID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" containerName="cleanup" Nov 03 20:15:24.849548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.849534 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" containerName="cleanup" Nov 03 20:15:24.849693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.849681 4457 cpu_manager.go:414] "containerMap: removing container" podUID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" containerName="cleanup" Nov 03 20:15:24.849782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.849770 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" containerName="cleanup" Nov 03 20:15:24.849871 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.849860 4457 cpu_manager.go:414] "containerMap: removing container" podUID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" containerName="cleanup" Nov 03 20:15:24.850002 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.849970 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" containerName="cleanup" Nov 03 20:15:24.850100 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.850087 4457 cpu_manager.go:414] "containerMap: removing container" podUID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" containerName="cleanup" Nov 03 20:15:24.850178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.850165 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" containerName="cleanup" Nov 03 20:15:24.850278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.850268 4457 cpu_manager.go:414] "containerMap: removing container" podUID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" containerName="cleanup" Nov 03 20:15:24.850360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.850349 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" containerName="cleanup" Nov 03 20:15:24.850525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.850510 4457 cpu_manager.go:414] "containerMap: removing container" podUID="00169716-105a-4f81-a435-bf13cfb20a8d" containerName="revoke-keys" Nov 03 20:15:24.850628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.850615 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00169716-105a-4f81-a435-bf13cfb20a8d" containerName="revoke-keys" Nov 03 20:15:24.850740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.850729 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c6fa0c0d-8a71-4ffa-ac05-04653af8b63c" containerName="revoke-keys" Nov 03 20:15:24.850831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.850818 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6fa0c0d-8a71-4ffa-ac05-04653af8b63c" containerName="revoke-keys" Nov 03 20:15:24.850937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.850925 4457 cpu_manager.go:414] "containerMap: removing container" podUID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" containerName="cleanup" Nov 03 20:15:24.851024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.851012 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" containerName="cleanup" Nov 03 20:15:24.851126 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.851115 4457 cpu_manager.go:414] "containerMap: removing container" podUID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" containerName="cleanup" Nov 03 20:15:24.851208 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.851196 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" containerName="cleanup" Nov 03 20:15:24.851691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.851649 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="00169716-105a-4f81-a435-bf13cfb20a8d" containerName="revoke-keys" Nov 03 20:15:24.851823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.851808 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" containerName="cleanup" Nov 03 20:15:24.851927 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.851914 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" containerName="cleanup" Nov 03 20:15:24.852083 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.852070 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6fa0c0d-8a71-4ffa-ac05-04653af8b63c" containerName="revoke-keys" Nov 03 20:15:24.852184 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.852173 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" containerName="cleanup" Nov 03 20:15:24.852272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.852262 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" containerName="cleanup" Nov 03 20:15:24.852355 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.852344 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81ab721-84a4-4df3-bcde-91a7b2c7cc8d" containerName="revoke-keys" Nov 03 20:15:24.852471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.852459 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" containerName="cleanup" Nov 03 20:15:24.852565 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.852555 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" containerName="cleanup" Nov 03 20:15:24.852649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.852639 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" containerName="cleanup" Nov 03 20:15:24.988562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:24.963922 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-6ab122" Nov 03 20:15:25.081415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.058189 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dwjcb\"" Nov 03 20:15:25.216413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.213453 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7084-6ab122"] Nov 03 20:15:25.228410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.219226 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv7ww\" (UniqueName: \"kubernetes.io/projected/0deb5216-b002-4ece-85b2-9d571a75e0a6-kube-api-access-qv7ww\") pod \"test-curl-7084-6ab122\" (UID: \"0deb5216-b002-4ece-85b2-9d571a75e0a6\") " pod="opendatahub/test-curl-7084-6ab122" Nov 03 20:15:25.412453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.412408 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv7ww\" (UniqueName: \"kubernetes.io/projected/0deb5216-b002-4ece-85b2-9d571a75e0a6-kube-api-access-qv7ww\") pod \"test-curl-7084-6ab122\" (UID: \"0deb5216-b002-4ece-85b2-9d571a75e0a6\") " pod="opendatahub/test-curl-7084-6ab122" Nov 03 20:15:25.490474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.490136 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv7ww\" (UniqueName: \"kubernetes.io/projected/0deb5216-b002-4ece-85b2-9d571a75e0a6-kube-api-access-qv7ww\") pod \"test-curl-7084-6ab122\" (UID: \"0deb5216-b002-4ece-85b2-9d571a75e0a6\") " pod="opendatahub/test-curl-7084-6ab122" Nov 03 20:15:25.515660 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.515616 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-6ab122" Nov 03 20:15:25.835420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.831870 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:25.835420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.832119 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:15:25.840320 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.838428 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 03 20:15:25.842470 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.840862 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:25.875501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.874498 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded" Nov 03 20:15:25.876108 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.876070 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:25.877705 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.877660 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:15:25.891298 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.880125 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:25.891298 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:25.890507 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.051686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.051641 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:15:26.108414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.106512 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.145764 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.145713 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.146472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.146021 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:15:26.147530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.146054 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:15:26.305474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.293778 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.473096 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.356491 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.789756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.782988 4457 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:15:26.861412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.835513 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:26.861412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.835589 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" podUID="94b5e918-acce-4fd9-ad4a-c37d87965667" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.861412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.838124 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.861412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.838462 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:15:26.861412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.838778 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:15:26.861412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.838837 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:15:26.861412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.840947 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.861412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.841263 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:15:26.861412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.859290 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.894714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.894489 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.894714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.894597 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.894714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.894650 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:26.909525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.909483 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:15:26.924474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.909800 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:15:26.924474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.909853 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:15:26.924474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.909901 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:15:26.924474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:26.909924 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7084-6ab122"] Nov 03 20:15:27.076432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.063004 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.116605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.116550 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 03 20:15:27.146355 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.120937 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:15:27.150413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.147721 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.205468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.150653 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.215413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.151161 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.244412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.202139 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:15:27.245171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.245143 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:15:27.245376 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.245363 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:15:27.304690 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.294818 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.304690 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.295351 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:15:27.322699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.322671 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:27.340578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.340535 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.343191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.343166 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:27.426315 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.426264 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.614440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.434090 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:15:27.631427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.622975 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:15:27.675372 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.674005 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.938937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.834417 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:27.938937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.834476 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.938937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.862923 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.938937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.867735 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.938937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.868992 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:15:27.955424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.940063 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:27.955424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.945502 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:15:27.955424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.946675 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:15:27.955424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:27.947660 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:15:28.072839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.072792 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:15:28.126752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.125643 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:28.245851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.245737 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:28.260414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.254495 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:15:28.269466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.265842 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:15:28.307409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.303378 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:28.355423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.354877 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:15:28.629265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.629187 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:28.651415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.650221 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:15:28.651415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.650358 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7084-6ab122" event={"ID":"0deb5216-b002-4ece-85b2-9d571a75e0a6","Type":"ContainerStarted","Data":"3f99d6455911a15045d342f2e9d5a7bd1024e28514e85fde6f5bf75e05785cd2"} Nov 03 20:15:28.651415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.650409 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k"] Nov 03 20:15:28.651415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.651347 4457 cpu_manager.go:414] "containerMap: removing container" podUID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" containerName="cleanup" Nov 03 20:15:28.651415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.651366 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" containerName="cleanup" Nov 03 20:15:28.651415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.651404 4457 cpu_manager.go:414] "containerMap: removing container" podUID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" containerName="cleanup" Nov 03 20:15:28.651415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.651413 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809babf5-bc1f-4a26-8fbd-67f1a5d0f11a" containerName="cleanup" Nov 03 20:15:28.651931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.651588 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6fe6d5-4a7f-46a1-b19c-f3ae1a17665c" containerName="cleanup" Nov 03 20:15:28.651931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.651603 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cc9e13f-05f6-4a9e-a15c-034c176b4695" containerName="cleanup" Nov 03 20:15:28.656513 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.656468 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:28.691424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.682843 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" Nov 03 20:15:28.691424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.691185 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:28.691676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.691482 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:15:28.707666 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.700832 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k"] Nov 03 20:15:28.809567 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.774113 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cccd04e9-n7z2k\" (UID: \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" Nov 03 20:15:28.809567 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.774440 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cccd04e9-n7z2k\" (UID: \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" Nov 03 20:15:28.809567 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.774529 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85mlx\" (UniqueName: \"kubernetes.io/projected/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-kube-api-access-85mlx\") pod \"maas-api-revoke-sub-cccd04e9-n7z2k\" (UID: \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" Nov 03 20:15:28.929551 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.908768 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cccd04e9-n7z2k\" (UID: \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" Nov 03 20:15:28.929551 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.908857 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85mlx\" (UniqueName: \"kubernetes.io/projected/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-kube-api-access-85mlx\") pod \"maas-api-revoke-sub-cccd04e9-n7z2k\" (UID: \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" Nov 03 20:15:28.929551 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.909082 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cccd04e9-n7z2k\" (UID: \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" Nov 03 20:15:28.929551 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.914416 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cccd04e9-n7z2k\" (UID: \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" Nov 03 20:15:28.950787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.930797 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 03 20:15:29.060839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.957726 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.060839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.958147 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:15:29.060839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.958446 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.060839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.958686 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:15:29.060839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.986843 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:15:29.060839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:28.987622 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:15:29.060839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.024196 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85mlx\" (UniqueName: \"kubernetes.io/projected/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-kube-api-access-85mlx\") pod \"maas-api-revoke-sub-cccd04e9-n7z2k\" (UID: \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" Nov 03 20:15:29.283541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.260104 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.301011 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.287184 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cccd04e9-n7z2k\" (UID: \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" Nov 03 20:15:29.338413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.302655 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" Nov 03 20:15:29.672509 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.384951 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:15:29.682539 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.682488 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:15:29.706413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.518325 4457 patch_prober.go:29] interesting pod/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:29.707064 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.656768 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:15:29.714740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.521374 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:15:29.714954 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.521517 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:15:29.715053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.521768 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.715162 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.521280 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:15:29.715322 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.385543 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.734478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.734436 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.734761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.734729 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:29.858626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:29.858589 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:15:30.005743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.005638 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 03 20:15:30.066128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.066065 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.329412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.309422 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 03 20:15:30.332241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.256699 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.402622 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.399537 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.501655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.501606 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:15:30.766903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.547430 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:15:30.767467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.547484 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:15:30.767815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.735559 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.799917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.798650 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:15:30.800334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.800196 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.825553 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.825500 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.826062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.825988 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:15:30.844415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.844350 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:30.903808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:30.894537 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:31.037521 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.031807 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:31.053837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.042240 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:31.102214 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:31.102178 4457 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.243s" Nov 03 20:15:31.102654 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.102635 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:15:31.102922 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.102908 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:15:31.103074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.103061 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:15:31.103229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.103218 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:15:31.115343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.115055 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:15:31.116676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.116658 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:15:31.116965 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.116942 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:15:31.131472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.131443 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:15:31.137411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.133116 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:15:31.137411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.133177 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:15:31.142143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.141959 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:15:31.152548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.152527 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:15:31.152798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.152774 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:15:31.153519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.153499 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:15:31.155369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.155354 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:15:31.325227 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:31.324657 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k"] Nov 03 20:15:32.018631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:32.018527 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" event={"ID":"91151417-29c0-40d5-80c2-bc8bb5bc5a1b","Type":"ContainerStarted","Data":"c96b763264634f7e7a269d1dca08c2f68ffc87340f16817e94191ee761b55619"} Nov 03 20:15:32.018631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:32.018593 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" event={"ID":"91151417-29c0-40d5-80c2-bc8bb5bc5a1b","Type":"ContainerStarted","Data":"dc78b5950923bfe2d500bea130d8fe0074f3511ea4cb4732048bb6b045189502"} Nov 03 20:15:32.039221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:32.039168 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" podStartSLOduration=5.039150255 podStartE2EDuration="5.039150255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:32.037229067 +0000 UTC m=+1110.412825814" watchObservedRunningTime="2026-11-03 20:15:32.039150255 +0000 UTC m=+1110.414747002" Nov 03 20:15:33.029831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:33.029792 4457 generic.go:356] "Generic (PLEG): container finished" podID="91151417-29c0-40d5-80c2-bc8bb5bc5a1b" containerID="c96b763264634f7e7a269d1dca08c2f68ffc87340f16817e94191ee761b55619" exitCode=0 Nov 03 20:15:33.030006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:33.029901 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" event={"ID":"91151417-29c0-40d5-80c2-bc8bb5bc5a1b","Type":"ContainerDied","Data":"c96b763264634f7e7a269d1dca08c2f68ffc87340f16817e94191ee761b55619"} Nov 03 20:15:34.045774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.045735 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7084-6ab122" event={"ID":"0deb5216-b002-4ece-85b2-9d571a75e0a6","Type":"ContainerStarted","Data":"218452b5f316c58c6554309b796bcea99980faa86172a6426a02171d32348000"} Nov 03 20:15:34.069894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.069810 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7084-6ab122" podStartSLOduration=3.431003236 podStartE2EDuration="10.069789672s" totalImagesPullingTime="6.638786436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:34.069694959 +0000 UTC m=+1112.445291772" watchObservedRunningTime="2026-11-03 20:15:34.069789672 +0000 UTC m=+1112.445386420" Nov 03 20:15:34.255503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.255476 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" Nov 03 20:15:34.348480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.348347 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-xks-ca\") pod \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\" (UID: \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\") " Nov 03 20:15:34.376136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.348655 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-openshift-ca\") pod \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\" (UID: \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\") " Nov 03 20:15:34.376136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.348684 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-kube-api-access-85mlx\" (UniqueName: \"kubernetes.io/projected/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-kube-api-access-85mlx\") pod \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\" (UID: \"91151417-29c0-40d5-80c2-bc8bb5bc5a1b\") " Nov 03 20:15:34.376136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.349027 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-openshift-ca" pod "91151417-29c0-40d5-80c2-bc8bb5bc5a1b" (UID: "91151417-29c0-40d5-80c2-bc8bb5bc5a1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:34.376136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.349280 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:34.376136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.350594 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-xks-ca" pod "91151417-29c0-40d5-80c2-bc8bb5bc5a1b" (UID: "91151417-29c0-40d5-80c2-bc8bb5bc5a1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:34.376136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.351009 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-kube-api-access-85mlx" pod "91151417-29c0-40d5-80c2-bc8bb5bc5a1b" (UID: "91151417-29c0-40d5-80c2-bc8bb5bc5a1b"). InnerVolumeSpecName "kube-api-access-85mlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:34.453484 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.453112 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:34.453484 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.453167 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85mlx\" (UniqueName: \"kubernetes.io/projected/91151417-29c0-40d5-80c2-bc8bb5bc5a1b-kube-api-access-85mlx\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:34.774917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.770839 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7084-6ab122"] Nov 03 20:15:34.781221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.781182 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7084-6ab122"] Nov 03 20:15:34.935689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.935646 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7084-4f7f8d"] Nov 03 20:15:34.936358 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.936343 4457 cpu_manager.go:414] "containerMap: removing container" podUID="0deb5216-b002-4ece-85b2-9d571a75e0a6" containerName="test-curl-7084-6ab122" Nov 03 20:15:34.936450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.936359 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0deb5216-b002-4ece-85b2-9d571a75e0a6" containerName="test-curl-7084-6ab122" Nov 03 20:15:34.936450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.936377 4457 cpu_manager.go:414] "containerMap: removing container" podUID="91151417-29c0-40d5-80c2-bc8bb5bc5a1b" containerName="revoke-keys" Nov 03 20:15:34.936450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.936395 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91151417-29c0-40d5-80c2-bc8bb5bc5a1b" containerName="revoke-keys" Nov 03 20:15:34.936549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.936504 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="0deb5216-b002-4ece-85b2-9d571a75e0a6" containerName="test-curl-7084-6ab122" Nov 03 20:15:34.936549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.936517 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="91151417-29c0-40d5-80c2-bc8bb5bc5a1b" containerName="revoke-keys" Nov 03 20:15:34.940485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.940456 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-4f7f8d" Nov 03 20:15:34.948894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:34.948191 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7084-4f7f8d"] Nov 03 20:15:35.056984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:35.056930 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" Nov 03 20:15:35.056984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:35.056958 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k" event={"ID":"91151417-29c0-40d5-80c2-bc8bb5bc5a1b","Type":"ContainerDied","Data":"dc78b5950923bfe2d500bea130d8fe0074f3511ea4cb4732048bb6b045189502"} Nov 03 20:15:35.057563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:35.057025 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc78b5950923bfe2d500bea130d8fe0074f3511ea4cb4732048bb6b045189502" Nov 03 20:15:35.063883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:35.063845 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ddv\" (UniqueName: \"kubernetes.io/projected/f539b160-b0d9-4f61-89c6-7b950adbfaef-kube-api-access-l7ddv\") pod \"test-curl-7084-4f7f8d\" (UID: \"f539b160-b0d9-4f61-89c6-7b950adbfaef\") " pod="opendatahub/test-curl-7084-4f7f8d" Nov 03 20:15:35.165695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:35.165657 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ddv\" (UniqueName: \"kubernetes.io/projected/f539b160-b0d9-4f61-89c6-7b950adbfaef-kube-api-access-l7ddv\") pod \"test-curl-7084-4f7f8d\" (UID: \"f539b160-b0d9-4f61-89c6-7b950adbfaef\") " pod="opendatahub/test-curl-7084-4f7f8d" Nov 03 20:15:35.178431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:35.178382 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ddv\" (UniqueName: \"kubernetes.io/projected/f539b160-b0d9-4f61-89c6-7b950adbfaef-kube-api-access-l7ddv\") pod \"test-curl-7084-4f7f8d\" (UID: \"f539b160-b0d9-4f61-89c6-7b950adbfaef\") " pod="opendatahub/test-curl-7084-4f7f8d" Nov 03 20:15:35.264695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:35.264658 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-4f7f8d" Nov 03 20:15:35.605488 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:15:35.605440 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf539b160_b0d9_4f61_89c6_7b950adbfaef.slice/crio-1c2f77e83a3794cd4a38ef3ed6ec6501be1d46440b957edc3b822c19854cffbc WatchSource:0}: Error finding container 1c2f77e83a3794cd4a38ef3ed6ec6501be1d46440b957edc3b822c19854cffbc: Status 404 returned error can't find the container with id 1c2f77e83a3794cd4a38ef3ed6ec6501be1d46440b957edc3b822c19854cffbc Nov 03 20:15:35.605639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:35.605518 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7084-4f7f8d"] Nov 03 20:15:36.069125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:36.069086 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7084-4f7f8d" event={"ID":"f539b160-b0d9-4f61-89c6-7b950adbfaef","Type":"ContainerStarted","Data":"3542d7c9af71e2167b5f902dda520d69f3ab1dc67d485a29cabcfe463148dc7f"} Nov 03 20:15:36.069125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:36.069136 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7084-4f7f8d" event={"ID":"f539b160-b0d9-4f61-89c6-7b950adbfaef","Type":"ContainerStarted","Data":"1c2f77e83a3794cd4a38ef3ed6ec6501be1d46440b957edc3b822c19854cffbc"} Nov 03 20:15:36.069712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:36.069518 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7084-6ab122" podUID="0deb5216-b002-4ece-85b2-9d571a75e0a6" containerName="test-curl-7084-6ab122" containerID="cri-o://218452b5f316c58c6554309b796bcea99980faa86172a6426a02171d32348000" gracePeriod=2 Nov 03 20:15:36.088575 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:36.088518 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7084-4f7f8d" podStartSLOduration=2.088503948 podStartE2EDuration="2.088503948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:36.087851142 +0000 UTC m=+1114.463447883" watchObservedRunningTime="2026-11-03 20:15:36.088503948 +0000 UTC m=+1114.464100692" Nov 03 20:15:36.186479 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:36.186437 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:15:36.854894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:36.854857 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7084-4f7f8d"] Nov 03 20:15:36.861708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:36.861672 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7084-4f7f8d"] Nov 03 20:15:38.100785 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.100720 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7084-4f7f8d" podUID="f539b160-b0d9-4f61-89c6-7b950adbfaef" containerName="test-curl-7084-4f7f8d" containerID="cri-o://3542d7c9af71e2167b5f902dda520d69f3ab1dc67d485a29cabcfe463148dc7f" gracePeriod=2 Nov 03 20:15:38.626667 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.617439 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv"] Nov 03 20:15:38.626667 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.618566 4457 cpu_manager.go:414] "containerMap: removing container" podUID="f539b160-b0d9-4f61-89c6-7b950adbfaef" containerName="test-curl-7084-4f7f8d" Nov 03 20:15:38.626667 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.618591 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f539b160-b0d9-4f61-89c6-7b950adbfaef" containerName="test-curl-7084-4f7f8d" Nov 03 20:15:38.626667 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.618760 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="f539b160-b0d9-4f61-89c6-7b950adbfaef" containerName="test-curl-7084-4f7f8d" Nov 03 20:15:38.639784 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.639747 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" Nov 03 20:15:38.647105 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.647014 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:38.647105 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.647066 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:15:38.653501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.653276 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv"] Nov 03 20:15:38.740839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.740684 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-6ab122" Nov 03 20:15:38.799402 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.799356 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58571cf8-lxztv\" (UID: \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" Nov 03 20:15:38.799598 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.799514 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58571cf8-lxztv\" (UID: \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" Nov 03 20:15:38.799705 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.799688 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8kqq\" (UniqueName: \"kubernetes.io/projected/0d059b46-7a38-40ab-9cca-cd51196e9c7e-kube-api-access-k8kqq\") pod \"maas-api-revoke-sub-58571cf8-lxztv\" (UID: \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" Nov 03 20:15:38.901167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.901062 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0deb5216-b002-4ece-85b2-9d571a75e0a6-kube-api-access-qv7ww\" (UniqueName: \"kubernetes.io/projected/0deb5216-b002-4ece-85b2-9d571a75e0a6-kube-api-access-qv7ww\") pod \"0deb5216-b002-4ece-85b2-9d571a75e0a6\" (UID: \"0deb5216-b002-4ece-85b2-9d571a75e0a6\") " Nov 03 20:15:38.901746 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.901717 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58571cf8-lxztv\" (UID: \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" Nov 03 20:15:38.901997 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.901977 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8kqq\" (UniqueName: \"kubernetes.io/projected/0d059b46-7a38-40ab-9cca-cd51196e9c7e-kube-api-access-k8kqq\") pod \"maas-api-revoke-sub-58571cf8-lxztv\" (UID: \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" Nov 03 20:15:38.902162 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.902071 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58571cf8-lxztv\" (UID: \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" Nov 03 20:15:38.902876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.902853 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58571cf8-lxztv\" (UID: \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" Nov 03 20:15:38.903603 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.903573 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0deb5216-b002-4ece-85b2-9d571a75e0a6-kube-api-access-qv7ww" pod "0deb5216-b002-4ece-85b2-9d571a75e0a6" (UID: "0deb5216-b002-4ece-85b2-9d571a75e0a6"). InnerVolumeSpecName "kube-api-access-qv7ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:38.904835 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.904808 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58571cf8-lxztv\" (UID: \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" Nov 03 20:15:38.914453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.914428 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8kqq\" (UniqueName: \"kubernetes.io/projected/0d059b46-7a38-40ab-9cca-cd51196e9c7e-kube-api-access-k8kqq\") pod \"maas-api-revoke-sub-58571cf8-lxztv\" (UID: \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" Nov 03 20:15:38.970382 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:38.970349 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" Nov 03 20:15:39.003613 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:39.003570 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv7ww\" (UniqueName: \"kubernetes.io/projected/0deb5216-b002-4ece-85b2-9d571a75e0a6-kube-api-access-qv7ww\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:39.112243 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:39.112188 4457 generic.go:356] "Generic (PLEG): container finished" podID="0deb5216-b002-4ece-85b2-9d571a75e0a6" containerID="218452b5f316c58c6554309b796bcea99980faa86172a6426a02171d32348000" exitCode=137 Nov 03 20:15:39.112727 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:39.112279 4457 scope.go:122] "RemoveContainer" containerID="218452b5f316c58c6554309b796bcea99980faa86172a6426a02171d32348000" Nov 03 20:15:39.112727 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:39.112523 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-6ab122" Nov 03 20:15:39.132231 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:39.132186 4457 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7084-6ab122\" is forbidden: User \"system:node:ip-10-0-132-25.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-25.us-east-2.compute.internal' and this object" podUID="0deb5216-b002-4ece-85b2-9d571a75e0a6" pod="opendatahub/test-curl-7084-6ab122" Nov 03 20:15:39.138027 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:39.138002 4457 scope.go:122] "RemoveContainer" containerID="218452b5f316c58c6554309b796bcea99980faa86172a6426a02171d32348000" Nov 03 20:15:39.138381 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:39.138349 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"218452b5f316c58c6554309b796bcea99980faa86172a6426a02171d32348000\": container with ID starting with 218452b5f316c58c6554309b796bcea99980faa86172a6426a02171d32348000 not found: ID does not exist" containerID="218452b5f316c58c6554309b796bcea99980faa86172a6426a02171d32348000" Nov 03 20:15:39.138381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:39.138406 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"218452b5f316c58c6554309b796bcea99980faa86172a6426a02171d32348000"} err="failed to get container status \"218452b5f316c58c6554309b796bcea99980faa86172a6426a02171d32348000\": rpc error: code = NotFound desc = could not find container \"218452b5f316c58c6554309b796bcea99980faa86172a6426a02171d32348000\": container with ID starting with 218452b5f316c58c6554309b796bcea99980faa86172a6426a02171d32348000 not found: ID does not exist" Nov 03 20:15:39.273351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:39.273322 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv"] Nov 03 20:15:39.794736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:39.793476 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0deb5216-b002-4ece-85b2-9d571a75e0a6" path="/var/lib/kubelet/pods/0deb5216-b002-4ece-85b2-9d571a75e0a6/volumes" Nov 03 20:15:40.133911 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:40.133869 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" event={"ID":"0d059b46-7a38-40ab-9cca-cd51196e9c7e","Type":"ContainerStarted","Data":"5137cd74fe509c2c61bcf508ffd51e64d6081b0e716cb3cee6793f5c383ccca2"} Nov 03 20:15:40.134356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:40.133941 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" event={"ID":"0d059b46-7a38-40ab-9cca-cd51196e9c7e","Type":"ContainerStarted","Data":"c8ca2d9a48293c3e799f74fb3bd4683c9bee1760a7690d54a256b8da98d2b1ac"} Nov 03 20:15:40.161110 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:40.161056 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" podStartSLOduration=2.16104126 podStartE2EDuration="2.16104126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:40.155958774 +0000 UTC m=+1118.531555537" watchObservedRunningTime="2026-11-03 20:15:40.16104126 +0000 UTC m=+1118.536638003" Nov 03 20:15:40.369054 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:40.368978 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-4f7f8d" Nov 03 20:15:40.545378 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:40.545333 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f539b160-b0d9-4f61-89c6-7b950adbfaef-kube-api-access-l7ddv\" (UniqueName: \"kubernetes.io/projected/f539b160-b0d9-4f61-89c6-7b950adbfaef-kube-api-access-l7ddv\") pod \"f539b160-b0d9-4f61-89c6-7b950adbfaef\" (UID: \"f539b160-b0d9-4f61-89c6-7b950adbfaef\") " Nov 03 20:15:40.547620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:40.547584 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f539b160-b0d9-4f61-89c6-7b950adbfaef-kube-api-access-l7ddv" pod "f539b160-b0d9-4f61-89c6-7b950adbfaef" (UID: "f539b160-b0d9-4f61-89c6-7b950adbfaef"). InnerVolumeSpecName "kube-api-access-l7ddv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:40.647753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:40.647679 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7ddv\" (UniqueName: \"kubernetes.io/projected/f539b160-b0d9-4f61-89c6-7b950adbfaef-kube-api-access-l7ddv\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:41.223416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:41.217654 4457 generic.go:356] "Generic (PLEG): container finished" podID="0d059b46-7a38-40ab-9cca-cd51196e9c7e" containerID="5137cd74fe509c2c61bcf508ffd51e64d6081b0e716cb3cee6793f5c383ccca2" exitCode=0 Nov 03 20:15:41.223416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:41.217873 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" event={"ID":"0d059b46-7a38-40ab-9cca-cd51196e9c7e","Type":"ContainerDied","Data":"5137cd74fe509c2c61bcf508ffd51e64d6081b0e716cb3cee6793f5c383ccca2"} Nov 03 20:15:41.228416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:41.227642 4457 generic.go:356] "Generic (PLEG): container finished" podID="f539b160-b0d9-4f61-89c6-7b950adbfaef" containerID="3542d7c9af71e2167b5f902dda520d69f3ab1dc67d485a29cabcfe463148dc7f" exitCode=137 Nov 03 20:15:41.228416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:41.227831 4457 scope.go:122] "RemoveContainer" containerID="3542d7c9af71e2167b5f902dda520d69f3ab1dc67d485a29cabcfe463148dc7f" Nov 03 20:15:41.228416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:41.228023 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7084-4f7f8d" Nov 03 20:15:41.341380 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:41.341330 4457 scope.go:122] "RemoveContainer" containerID="3542d7c9af71e2167b5f902dda520d69f3ab1dc67d485a29cabcfe463148dc7f" Nov 03 20:15:41.350003 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:41.349950 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3542d7c9af71e2167b5f902dda520d69f3ab1dc67d485a29cabcfe463148dc7f\": container with ID starting with 3542d7c9af71e2167b5f902dda520d69f3ab1dc67d485a29cabcfe463148dc7f not found: ID does not exist" containerID="3542d7c9af71e2167b5f902dda520d69f3ab1dc67d485a29cabcfe463148dc7f" Nov 03 20:15:41.350249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:41.350222 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3542d7c9af71e2167b5f902dda520d69f3ab1dc67d485a29cabcfe463148dc7f"} err="failed to get container status \"3542d7c9af71e2167b5f902dda520d69f3ab1dc67d485a29cabcfe463148dc7f\": rpc error: code = NotFound desc = could not find container \"3542d7c9af71e2167b5f902dda520d69f3ab1dc67d485a29cabcfe463148dc7f\": container with ID starting with 3542d7c9af71e2167b5f902dda520d69f3ab1dc67d485a29cabcfe463148dc7f not found: ID does not exist" Nov 03 20:15:41.353153 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:41.353116 4457 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7084-4f7f8d\" is forbidden: User \"system:node:ip-10-0-132-25.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-25.us-east-2.compute.internal' and this object" podUID="f539b160-b0d9-4f61-89c6-7b950adbfaef" pod="opendatahub/test-curl-7084-4f7f8d" Nov 03 20:15:41.758434 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:15:41.758375 4457 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7084-4f7f8d\" is forbidden: User \"system:node:ip-10-0-132-25.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-25.us-east-2.compute.internal' and this object" podUID="f539b160-b0d9-4f61-89c6-7b950adbfaef" pod="opendatahub/test-curl-7084-4f7f8d" Nov 03 20:15:41.762903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:41.762870 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f539b160-b0d9-4f61-89c6-7b950adbfaef" path="/var/lib/kubelet/pods/f539b160-b0d9-4f61-89c6-7b950adbfaef/volumes" Nov 03 20:15:42.522672 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.522593 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" Nov 03 20:15:42.622188 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.622150 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-openshift-ca\") pod \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\" (UID: \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\") " Nov 03 20:15:42.623051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.623022 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-xks-ca\") pod \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\" (UID: \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\") " Nov 03 20:15:42.623190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.623121 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d059b46-7a38-40ab-9cca-cd51196e9c7e-kube-api-access-k8kqq\" (UniqueName: \"kubernetes.io/projected/0d059b46-7a38-40ab-9cca-cd51196e9c7e-kube-api-access-k8kqq\") pod \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\" (UID: \"0d059b46-7a38-40ab-9cca-cd51196e9c7e\") " Nov 03 20:15:42.623953 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.622769 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-openshift-ca" pod "0d059b46-7a38-40ab-9cca-cd51196e9c7e" (UID: "0d059b46-7a38-40ab-9cca-cd51196e9c7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:42.626226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.626202 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d059b46-7a38-40ab-9cca-cd51196e9c7e-kube-api-access-k8kqq" pod "0d059b46-7a38-40ab-9cca-cd51196e9c7e" (UID: "0d059b46-7a38-40ab-9cca-cd51196e9c7e"). InnerVolumeSpecName "kube-api-access-k8kqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:42.634694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.634542 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-xks-ca" pod "0d059b46-7a38-40ab-9cca-cd51196e9c7e" (UID: "0d059b46-7a38-40ab-9cca-cd51196e9c7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:42.725165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.725125 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:42.725165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.725161 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8kqq\" (UniqueName: \"kubernetes.io/projected/0d059b46-7a38-40ab-9cca-cd51196e9c7e-kube-api-access-k8kqq\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:42.725379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.725175 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d059b46-7a38-40ab-9cca-cd51196e9c7e-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:42.897507 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.897472 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj"] Nov 03 20:15:42.898253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.898232 4457 cpu_manager.go:414] "containerMap: removing container" podUID="0d059b46-7a38-40ab-9cca-cd51196e9c7e" containerName="revoke-keys" Nov 03 20:15:42.898363 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.898254 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d059b46-7a38-40ab-9cca-cd51196e9c7e" containerName="revoke-keys" Nov 03 20:15:42.898473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.898422 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d059b46-7a38-40ab-9cca-cd51196e9c7e" containerName="revoke-keys" Nov 03 20:15:42.915400 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.915358 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p"] Nov 03 20:15:42.915792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.915763 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" Nov 03 20:15:42.931862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.927217 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0416e8f1-dockercfg-cgmdp\"" Nov 03 20:15:42.931862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.927869 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0416e8f1\"" Nov 03 20:15:42.931862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.927949 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d3f423c-4451-4f01-8be1-44bf84c13e9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj\" (UID: \"9d3f423c-4451-4f01-8be1-44bf84c13e9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" Nov 03 20:15:42.931862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.928026 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj9bq\" (UniqueName: \"kubernetes.io/projected/9d3f423c-4451-4f01-8be1-44bf84c13e9a-kube-api-access-gj9bq\") pod \"payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj\" (UID: \"9d3f423c-4451-4f01-8be1-44bf84c13e9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" Nov 03 20:15:42.942457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.942381 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj"] Nov 03 20:15:42.942629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.942561 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" Nov 03 20:15:42.949742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:42.949696 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p"] Nov 03 20:15:43.030210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.030174 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljndd\" (UniqueName: \"kubernetes.io/projected/79caffd6-766b-4a72-951d-901ee5497282-kube-api-access-ljndd\") pod \"payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p\" (UID: \"79caffd6-766b-4a72-951d-901ee5497282\") " pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" Nov 03 20:15:43.030382 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.030268 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d3f423c-4451-4f01-8be1-44bf84c13e9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj\" (UID: \"9d3f423c-4451-4f01-8be1-44bf84c13e9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" Nov 03 20:15:43.030382 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.030338 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9bq\" (UniqueName: \"kubernetes.io/projected/9d3f423c-4451-4f01-8be1-44bf84c13e9a-kube-api-access-gj9bq\") pod \"payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj\" (UID: \"9d3f423c-4451-4f01-8be1-44bf84c13e9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" Nov 03 20:15:43.030578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.030558 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79caffd6-766b-4a72-951d-901ee5497282-plugins-config-volume\") pod \"payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p\" (UID: \"79caffd6-766b-4a72-951d-901ee5497282\") " pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" Nov 03 20:15:43.031265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.031238 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d3f423c-4451-4f01-8be1-44bf84c13e9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj\" (UID: \"9d3f423c-4451-4f01-8be1-44bf84c13e9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" Nov 03 20:15:43.084410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.083732 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9bq\" (UniqueName: \"kubernetes.io/projected/9d3f423c-4451-4f01-8be1-44bf84c13e9a-kube-api-access-gj9bq\") pod \"payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj\" (UID: \"9d3f423c-4451-4f01-8be1-44bf84c13e9a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" Nov 03 20:15:43.134024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.133986 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79caffd6-766b-4a72-951d-901ee5497282-plugins-config-volume\") pod \"payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p\" (UID: \"79caffd6-766b-4a72-951d-901ee5497282\") " pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" Nov 03 20:15:43.139701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.139670 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljndd\" (UniqueName: \"kubernetes.io/projected/79caffd6-766b-4a72-951d-901ee5497282-kube-api-access-ljndd\") pod \"payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p\" (UID: \"79caffd6-766b-4a72-951d-901ee5497282\") " pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" Nov 03 20:15:43.141047 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.135085 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79caffd6-766b-4a72-951d-901ee5497282-plugins-config-volume\") pod \"payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p\" (UID: \"79caffd6-766b-4a72-951d-901ee5497282\") " pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" Nov 03 20:15:43.195276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.195178 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljndd\" (UniqueName: \"kubernetes.io/projected/79caffd6-766b-4a72-951d-901ee5497282-kube-api-access-ljndd\") pod \"payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p\" (UID: \"79caffd6-766b-4a72-951d-901ee5497282\") " pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" Nov 03 20:15:43.245090 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.245039 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" Nov 03 20:15:43.326410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.266202 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" Nov 03 20:15:43.358920 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.358885 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" event={"ID":"0d059b46-7a38-40ab-9cca-cd51196e9c7e","Type":"ContainerDied","Data":"c8ca2d9a48293c3e799f74fb3bd4683c9bee1760a7690d54a256b8da98d2b1ac"} Nov 03 20:15:43.359564 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.359542 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8ca2d9a48293c3e799f74fb3bd4683c9bee1760a7690d54a256b8da98d2b1ac" Nov 03 20:15:43.359776 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.359763 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv" Nov 03 20:15:43.757913 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:15:43.757805 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3f423c_4451_4f01_8be1_44bf84c13e9a.slice/crio-029f3d9d107598e10adad411a9d592c186c397930b9432db50973266b4aa9467 WatchSource:0}: Error finding container 029f3d9d107598e10adad411a9d592c186c397930b9432db50973266b4aa9467: Status 404 returned error can't find the container with id 029f3d9d107598e10adad411a9d592c186c397930b9432db50973266b4aa9467 Nov 03 20:15:43.764484 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.764457 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj"] Nov 03 20:15:43.768087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:43.768048 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p"] Nov 03 20:15:44.007329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.007290 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz"] Nov 03 20:15:44.013296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.013223 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" Nov 03 20:15:44.013861 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.013830 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz"] Nov 03 20:15:44.018656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.018629 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:44.019286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.019264 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:15:44.091607 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.091567 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad10b3a-gzhqz\" (UID: \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" Nov 03 20:15:44.091607 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.091621 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad10b3a-gzhqz\" (UID: \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" Nov 03 20:15:44.091927 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.091900 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxlqr\" (UniqueName: \"kubernetes.io/projected/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-kube-api-access-zxlqr\") pod \"maas-api-revoke-sub-aad10b3a-gzhqz\" (UID: \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" Nov 03 20:15:44.193016 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.192984 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxlqr\" (UniqueName: \"kubernetes.io/projected/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-kube-api-access-zxlqr\") pod \"maas-api-revoke-sub-aad10b3a-gzhqz\" (UID: \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" Nov 03 20:15:44.193203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.193178 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad10b3a-gzhqz\" (UID: \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" Nov 03 20:15:44.193278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.193218 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad10b3a-gzhqz\" (UID: \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" Nov 03 20:15:44.193977 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.193953 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad10b3a-gzhqz\" (UID: \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" Nov 03 20:15:44.195664 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.195644 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad10b3a-gzhqz\" (UID: \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" Nov 03 20:15:44.205398 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.205366 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxlqr\" (UniqueName: \"kubernetes.io/projected/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-kube-api-access-zxlqr\") pod \"maas-api-revoke-sub-aad10b3a-gzhqz\" (UID: \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" Nov 03 20:15:44.333400 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.333366 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" Nov 03 20:15:44.370907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.370857 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" event={"ID":"9d3f423c-4451-4f01-8be1-44bf84c13e9a","Type":"ContainerStarted","Data":"5c0bf30ee9c2030f1ef350061b9890ff9c4456b2122f1a81d44ac02bc9dbf5af"} Nov 03 20:15:44.371082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.370931 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" event={"ID":"9d3f423c-4451-4f01-8be1-44bf84c13e9a","Type":"ContainerStarted","Data":"029f3d9d107598e10adad411a9d592c186c397930b9432db50973266b4aa9467"} Nov 03 20:15:44.371082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.370955 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" Nov 03 20:15:44.373595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.373565 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" event={"ID":"79caffd6-766b-4a72-951d-901ee5497282","Type":"ContainerStarted","Data":"634fbca07ae8e899ab5019bb147eb57b1395b993ee835de8255a9abd13419aa8"} Nov 03 20:15:44.373764 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.373650 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" event={"ID":"79caffd6-766b-4a72-951d-901ee5497282","Type":"ContainerStarted","Data":"1a2867f5426579ab58843bf2fc0eb2da793c0f7f25404d102c5a9b2b5b558465"} Nov 03 20:15:44.373764 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.373678 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" Nov 03 20:15:44.405261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.405211 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" podStartSLOduration=2.405195603 podStartE2EDuration="2.405195603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:44.400482062 +0000 UTC m=+1122.776078813" watchObservedRunningTime="2026-11-03 20:15:44.405195603 +0000 UTC m=+1122.780792349" Nov 03 20:15:44.445735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.445673 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" podStartSLOduration=2.445654058 podStartE2EDuration="2.445654058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:44.445529724 +0000 UTC m=+1122.821126496" watchObservedRunningTime="2026-11-03 20:15:44.445654058 +0000 UTC m=+1122.821250806" Nov 03 20:15:44.645171 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:15:44.645136 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb24461a_5b4c_4c13_b817_c2c7f3a02b94.slice/crio-4182a4f5b3e2066b94a1d2ca8f358e905e328ad8996ed446dfea4ab121b0ed10 WatchSource:0}: Error finding container 4182a4f5b3e2066b94a1d2ca8f358e905e328ad8996ed446dfea4ab121b0ed10: Status 404 returned error can't find the container with id 4182a4f5b3e2066b94a1d2ca8f358e905e328ad8996ed446dfea4ab121b0ed10 Nov 03 20:15:44.645843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:44.645809 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz"] Nov 03 20:15:45.384774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:45.384736 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" event={"ID":"eb24461a-5b4c-4c13-b817-c2c7f3a02b94","Type":"ContainerStarted","Data":"b94d1811087b324df1941ac2fe882ecd924afc5f15254a7d7f3b114e2cee7237"} Nov 03 20:15:45.385237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:45.384802 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" event={"ID":"eb24461a-5b4c-4c13-b817-c2c7f3a02b94","Type":"ContainerStarted","Data":"4182a4f5b3e2066b94a1d2ca8f358e905e328ad8996ed446dfea4ab121b0ed10"} Nov 03 20:15:45.416580 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:45.416521 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" podStartSLOduration=2.416505599 podStartE2EDuration="2.416505599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:45.415120188 +0000 UTC m=+1123.790716949" watchObservedRunningTime="2026-11-03 20:15:45.416505599 +0000 UTC m=+1123.792102336" Nov 03 20:15:46.405137 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:46.405097 4457 generic.go:356] "Generic (PLEG): container finished" podID="eb24461a-5b4c-4c13-b817-c2c7f3a02b94" containerID="b94d1811087b324df1941ac2fe882ecd924afc5f15254a7d7f3b114e2cee7237" exitCode=0 Nov 03 20:15:46.407496 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:46.405189 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" event={"ID":"eb24461a-5b4c-4c13-b817-c2c7f3a02b94","Type":"ContainerDied","Data":"b94d1811087b324df1941ac2fe882ecd924afc5f15254a7d7f3b114e2cee7237"} Nov 03 20:15:47.143326 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.143283 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h"] Nov 03 20:15:47.149458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.149429 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.169591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.169544 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-e44e2278-gw-openshift-default-dockercfg-l56vz\"" Nov 03 20:15:47.183473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.181519 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h"] Nov 03 20:15:47.349579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.328824 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istiod-ca-cert\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.349579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.328951 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-envoy\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.349579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.329019 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-certs\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.349579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.329058 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-socket\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.349579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.329094 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-credential-socket\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.349579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.329263 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-token\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.349579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.329341 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqmzj\" (UniqueName: \"kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-kube-api-access-xqmzj\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.349579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.329437 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-podinfo\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.349579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.329485 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-data\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.450676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.447794 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-token\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.450676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.447889 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqmzj\" (UniqueName: \"kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-kube-api-access-xqmzj\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.450676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.447955 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-podinfo\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.450676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.447998 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-data\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.450676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.448068 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istiod-ca-cert\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.450676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.448160 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-envoy\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.450676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.448219 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-certs\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.450676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.448257 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-socket\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.450676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.448289 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-credential-socket\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.450676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.448878 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-credential-socket\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.501883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.499974 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-data\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.501883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.500668 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istiod-ca-cert\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.525413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.522665 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-certs\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.525413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.522913 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-socket\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.538681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.534143 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqmzj\" (UniqueName: \"kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-kube-api-access-xqmzj\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.538681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.535295 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-envoy\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.538681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.535590 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-token\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.538681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.537663 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-podinfo\") pod \"e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.778425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.773042 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:47.800292 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.800253 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" Nov 03 20:15:47.868946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.868915 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-kube-api-access-zxlqr\" (UniqueName: \"kubernetes.io/projected/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-kube-api-access-zxlqr\") pod \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\" (UID: \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\") " Nov 03 20:15:47.869101 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.868991 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-xks-ca\") pod \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\" (UID: \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\") " Nov 03 20:15:47.871079 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.869174 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-openshift-ca\") pod \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\" (UID: \"eb24461a-5b4c-4c13-b817-c2c7f3a02b94\") " Nov 03 20:15:47.871079 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.870954 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-openshift-ca" pod "eb24461a-5b4c-4c13-b817-c2c7f3a02b94" (UID: "eb24461a-5b4c-4c13-b817-c2c7f3a02b94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:47.880530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.880491 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-xks-ca" pod "eb24461a-5b4c-4c13-b817-c2c7f3a02b94" (UID: "eb24461a-5b4c-4c13-b817-c2c7f3a02b94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:47.881043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.881012 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-kube-api-access-zxlqr" pod "eb24461a-5b4c-4c13-b817-c2c7f3a02b94" (UID: "eb24461a-5b4c-4c13-b817-c2c7f3a02b94"). InnerVolumeSpecName "kube-api-access-zxlqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:47.976189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.975082 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:47.976189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.975119 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxlqr\" (UniqueName: \"kubernetes.io/projected/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-kube-api-access-zxlqr\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:47.976189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:47.975135 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb24461a-5b4c-4c13-b817-c2c7f3a02b94-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:48.117149 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:48.117114 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h"] Nov 03 20:15:48.117370 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:48.117338 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:48.117536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:48.117517 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:48.117623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:48.117613 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:15:48.572412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:48.570703 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" event={"ID":"eb24461a-5b4c-4c13-b817-c2c7f3a02b94","Type":"ContainerDied","Data":"4182a4f5b3e2066b94a1d2ca8f358e905e328ad8996ed446dfea4ab121b0ed10"} Nov 03 20:15:48.572412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:48.570799 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4182a4f5b3e2066b94a1d2ca8f358e905e328ad8996ed446dfea4ab121b0ed10" Nov 03 20:15:48.572412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:48.570894 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz" Nov 03 20:15:48.579863 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:48.579825 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" event={"ID":"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd","Type":"ContainerStarted","Data":"9d6a14f62577806887dae01e44049d82d94da6a70ba099ecc7eb853dbb4c1818"} Nov 03 20:15:48.580013 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:48.579892 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" event={"ID":"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd","Type":"ContainerStarted","Data":"4c24f7cfe2516909e60910e1a71d5c06efb461922ceff1b07ce35c2d407cd2e6"} Nov 03 20:15:48.609261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:48.609148 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" podStartSLOduration=1.6091298219999999 podStartE2EDuration="1.609129822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:48.603454178 +0000 UTC m=+1126.979050927" watchObservedRunningTime="2026-11-03 20:15:48.609129822 +0000 UTC m=+1126.984726570" Nov 03 20:15:48.775475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:48.773962 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:49.781839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:49.781802 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:50.601093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:50.601059 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:50.603174 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:50.603148 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:15:53.239922 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.239884 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl"] Nov 03 20:15:53.240633 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.240615 4457 cpu_manager.go:414] "containerMap: removing container" podUID="eb24461a-5b4c-4c13-b817-c2c7f3a02b94" containerName="revoke-keys" Nov 03 20:15:53.240633 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.240634 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb24461a-5b4c-4c13-b817-c2c7f3a02b94" containerName="revoke-keys" Nov 03 20:15:53.240774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.240762 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb24461a-5b4c-4c13-b817-c2c7f3a02b94" containerName="revoke-keys" Nov 03 20:15:53.244737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.244706 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" Nov 03 20:15:53.251184 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.251155 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:15:53.252605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.252584 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:15:53.253400 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.253364 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl"] Nov 03 20:15:53.434003 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.433961 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb78q\" (UniqueName: \"kubernetes.io/projected/33a0479f-d3fd-4919-9bac-d69caaa62449-kube-api-access-lb78q\") pod \"maas-api-revoke-sub-9886285f-sx9vl\" (UID: \"33a0479f-d3fd-4919-9bac-d69caaa62449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" Nov 03 20:15:53.434192 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.434007 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9886285f-sx9vl\" (UID: \"33a0479f-d3fd-4919-9bac-d69caaa62449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" Nov 03 20:15:53.434236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.434207 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9886285f-sx9vl\" (UID: \"33a0479f-d3fd-4919-9bac-d69caaa62449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" Nov 03 20:15:53.536295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.536205 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9886285f-sx9vl\" (UID: \"33a0479f-d3fd-4919-9bac-d69caaa62449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" Nov 03 20:15:53.536549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.536520 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb78q\" (UniqueName: \"kubernetes.io/projected/33a0479f-d3fd-4919-9bac-d69caaa62449-kube-api-access-lb78q\") pod \"maas-api-revoke-sub-9886285f-sx9vl\" (UID: \"33a0479f-d3fd-4919-9bac-d69caaa62449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" Nov 03 20:15:53.537094 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.536559 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9886285f-sx9vl\" (UID: \"33a0479f-d3fd-4919-9bac-d69caaa62449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" Nov 03 20:15:53.537331 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.537312 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9886285f-sx9vl\" (UID: \"33a0479f-d3fd-4919-9bac-d69caaa62449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" Nov 03 20:15:53.538735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.538711 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9886285f-sx9vl\" (UID: \"33a0479f-d3fd-4919-9bac-d69caaa62449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" Nov 03 20:15:53.549399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.549359 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb78q\" (UniqueName: \"kubernetes.io/projected/33a0479f-d3fd-4919-9bac-d69caaa62449-kube-api-access-lb78q\") pod \"maas-api-revoke-sub-9886285f-sx9vl\" (UID: \"33a0479f-d3fd-4919-9bac-d69caaa62449\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" Nov 03 20:15:53.585221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.585186 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" Nov 03 20:15:53.916059 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:15:53.915996 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a0479f_d3fd_4919_9bac_d69caaa62449.slice/crio-faa16139e11797f3f0ac0b9fab9afde75d959285e220556ac309e35e7a771cf6 WatchSource:0}: Error finding container faa16139e11797f3f0ac0b9fab9afde75d959285e220556ac309e35e7a771cf6: Status 404 returned error can't find the container with id faa16139e11797f3f0ac0b9fab9afde75d959285e220556ac309e35e7a771cf6 Nov 03 20:15:53.917132 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:53.917098 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl"] Nov 03 20:15:54.343210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.343168 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz"] Nov 03 20:15:54.364305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.354545 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.364305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.363840 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz"] Nov 03 20:15:54.364933 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.364903 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c847b846-serving-cert\"" Nov 03 20:15:54.455041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.454957 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-maas-api-tls\") pod \"maas-api-e2e-ait-c847b846-98c4694d7-kk7gz\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.455186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.455074 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnwnl\" (UniqueName: \"kubernetes.io/projected/8a708372-3610-4b08-b671-c76e6ef46675-kube-api-access-rnwnl\") pod \"maas-api-e2e-ait-c847b846-98c4694d7-kk7gz\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.455186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.455137 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a708372-3610-4b08-b671-c76e6ef46675-openshift-service-ca\") pod \"maas-api-e2e-ait-c847b846-98c4694d7-kk7gz\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.455186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.455163 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-metrics-certs\") pod \"maas-api-e2e-ait-c847b846-98c4694d7-kk7gz\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.583973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.583888 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-maas-api-tls\") pod \"maas-api-e2e-ait-c847b846-98c4694d7-kk7gz\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.584181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.583993 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnwnl\" (UniqueName: \"kubernetes.io/projected/8a708372-3610-4b08-b671-c76e6ef46675-kube-api-access-rnwnl\") pod \"maas-api-e2e-ait-c847b846-98c4694d7-kk7gz\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.584181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.584048 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a708372-3610-4b08-b671-c76e6ef46675-openshift-service-ca\") pod \"maas-api-e2e-ait-c847b846-98c4694d7-kk7gz\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.584181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.584086 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-metrics-certs\") pod \"maas-api-e2e-ait-c847b846-98c4694d7-kk7gz\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.590401 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.590354 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-metrics-certs\") pod \"maas-api-e2e-ait-c847b846-98c4694d7-kk7gz\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.600740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.600696 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a708372-3610-4b08-b671-c76e6ef46675-openshift-service-ca\") pod \"maas-api-e2e-ait-c847b846-98c4694d7-kk7gz\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.616421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.615117 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-maas-api-tls\") pod \"maas-api-e2e-ait-c847b846-98c4694d7-kk7gz\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.617341 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.617311 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnwnl\" (UniqueName: \"kubernetes.io/projected/8a708372-3610-4b08-b671-c76e6ef46675-kube-api-access-rnwnl\") pod \"maas-api-e2e-ait-c847b846-98c4694d7-kk7gz\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.669273 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.669224 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" event={"ID":"33a0479f-d3fd-4919-9bac-d69caaa62449","Type":"ContainerStarted","Data":"479d2d78019fc8eeca15bd6f4d4eb0b547fe8e9191a61a3bda3fedb56b956aa6"} Nov 03 20:15:54.669495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.669296 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" event={"ID":"33a0479f-d3fd-4919-9bac-d69caaa62449","Type":"ContainerStarted","Data":"faa16139e11797f3f0ac0b9fab9afde75d959285e220556ac309e35e7a771cf6"} Nov 03 20:15:54.691625 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.691585 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:54.708536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:54.707353 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" podStartSLOduration=1.7073311009999999 podStartE2EDuration="1.707331101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:54.689637853 +0000 UTC m=+1133.065234593" watchObservedRunningTime="2026-11-03 20:15:54.707331101 +0000 UTC m=+1133.082927864" Nov 03 20:15:55.085411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:55.082554 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz"] Nov 03 20:15:55.388272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:55.388193 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" Nov 03 20:15:55.388740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:55.388375 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" Nov 03 20:15:55.732726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:55.732692 4457 generic.go:356] "Generic (PLEG): container finished" podID="33a0479f-d3fd-4919-9bac-d69caaa62449" containerID="479d2d78019fc8eeca15bd6f4d4eb0b547fe8e9191a61a3bda3fedb56b956aa6" exitCode=0 Nov 03 20:15:55.732930 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:55.732806 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" event={"ID":"33a0479f-d3fd-4919-9bac-d69caaa62449","Type":"ContainerDied","Data":"479d2d78019fc8eeca15bd6f4d4eb0b547fe8e9191a61a3bda3fedb56b956aa6"} Nov 03 20:15:55.738898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:55.738795 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" event={"ID":"8a708372-3610-4b08-b671-c76e6ef46675","Type":"ContainerStarted","Data":"af40a0c6c9aeb63157d059ed3f4b2afef4453d17b3e5341c1dfa34906384a244"} Nov 03 20:15:56.813069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:56.813024 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" event={"ID":"8a708372-3610-4b08-b671-c76e6ef46675","Type":"ContainerStarted","Data":"eaecee9ddadec2e2e4625987fd363466fc40a2cb2be2aaa9bf37a62ff8ed0492"} Nov 03 20:15:56.825678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:56.813108 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:15:56.943442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:56.939759 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" podStartSLOduration=2.537118231 podStartE2EDuration="2.93973491s" totalImagesPullingTime="402.616679ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:15:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:15:56.885746544 +0000 UTC m=+1135.261343292" watchObservedRunningTime="2026-11-03 20:15:56.93973491 +0000 UTC m=+1135.315331658" Nov 03 20:15:57.776751 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:57.756675 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:57.802413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:57.797538 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:57.910412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:57.909991 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.013410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:57.973692 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.085415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.017471 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.210440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.164650 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:15:58.210440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.198934 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:15:58.210440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.202464 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:15:58.264689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.264614 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.275864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.275801 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.296152 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.296114 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" Nov 03 20:15:58.307539 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.297491 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.308572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.308523 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.469578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.469472 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-openshift-ca\") pod \"33a0479f-d3fd-4919-9bac-d69caaa62449\" (UID: \"33a0479f-d3fd-4919-9bac-d69caaa62449\") " Nov 03 20:15:58.470334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.470307 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-xks-ca\") pod \"33a0479f-d3fd-4919-9bac-d69caaa62449\" (UID: \"33a0479f-d3fd-4919-9bac-d69caaa62449\") " Nov 03 20:15:58.471017 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.470995 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a0479f-d3fd-4919-9bac-d69caaa62449-kube-api-access-lb78q\" (UniqueName: \"kubernetes.io/projected/33a0479f-d3fd-4919-9bac-d69caaa62449-kube-api-access-lb78q\") pod \"33a0479f-d3fd-4919-9bac-d69caaa62449\" (UID: \"33a0479f-d3fd-4919-9bac-d69caaa62449\") " Nov 03 20:15:58.472337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.470198 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-openshift-ca" pod "33a0479f-d3fd-4919-9bac-d69caaa62449" (UID: "33a0479f-d3fd-4919-9bac-d69caaa62449"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:15:58.519279 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.519225 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-xks-ca" pod "33a0479f-d3fd-4919-9bac-d69caaa62449" (UID: "33a0479f-d3fd-4919-9bac-d69caaa62449"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:15:58.541331 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.541294 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a0479f-d3fd-4919-9bac-d69caaa62449-kube-api-access-lb78q" pod "33a0479f-d3fd-4919-9bac-d69caaa62449" (UID: "33a0479f-d3fd-4919-9bac-d69caaa62449"). InnerVolumeSpecName "kube-api-access-lb78q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:15:58.580693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.580653 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:58.581369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.581348 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33a0479f-d3fd-4919-9bac-d69caaa62449-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:58.581530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.581513 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb78q\" (UniqueName: \"kubernetes.io/projected/33a0479f-d3fd-4919-9bac-d69caaa62449-kube-api-access-lb78q\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:15:58.655309 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.655043 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:58.847433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.839515 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:15:58.847433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.846472 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:15:58.878432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:58.864888 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:15:59.232488 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.230761 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:15:59.235487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.233873 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:15:59.262480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.258552 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:15:59.262480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.258945 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.355166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.355110 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.355922 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.355892 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:15:59.585448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.373544 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:15:59.585448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.404354 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:15:59.585448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.559000 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:15:59.585448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.559131 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:15:59.585448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.559278 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:15:59.738521 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.738466 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.985934 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.985811 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:15:59.992089 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:15:59.992050 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.020503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.020447 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.021289 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.021255 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.022067 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.022041 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.022370 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.022344 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 03 20:16:00.115420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.043375 4457 patch_prober.go:29] interesting pod/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:00.115420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.043462 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.115420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.047322 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:16:00.240478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.236725 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:16:00.429425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.393064 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": context deadline exceeded" Nov 03 20:16:00.532001 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.416136 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.532612 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.416164 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:16:00.542420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.416082 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.806412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.417380 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:00.832605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.832549 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.868532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.558538 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.895370 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.755608 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:16:00.929007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.928965 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" Nov 03 20:16:00.929495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.558497 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.930092 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.791502 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:00.930290 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.930255 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:00.931108 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.810539 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:16:00.956677 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.956616 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl" event={"ID":"33a0479f-d3fd-4919-9bac-d69caaa62449","Type":"ContainerDied","Data":"faa16139e11797f3f0ac0b9fab9afde75d959285e220556ac309e35e7a771cf6"} Nov 03 20:16:00.959441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.959415 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faa16139e11797f3f0ac0b9fab9afde75d959285e220556ac309e35e7a771cf6" Nov 03 20:16:00.959744 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.959726 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:16:00.959911 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.959898 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:16:00.960078 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.960058 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:16:00.960221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.960209 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:16:00.960362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:00.960350 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:16:01.046698 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.046662 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:01.063687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.063592 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 03 20:16:01.105452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.073638 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:16:01.106817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.095079 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:16:01.487527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.095120 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:16:01.503299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.095142 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:16:01.503299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.095162 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:16:01.503299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.095201 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:16:01.503299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.212863 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:16:01.503299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.243501 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:01.503299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.294590 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:01.503299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.503231 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:01.535732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.535623 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:01.541374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.538435 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:16:01.541374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.540835 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:16:01.550552 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.548933 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:16:01.585671 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.583907 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:01.585671 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.585043 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:01.585671 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.585094 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" podUID="49cbfb85-a563-4265-828a-40ead144734b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:01.668424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.649418 4457 patch_prober.go:29] interesting pod/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:01.668728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.668685 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:01.882296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.882163 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:01.909501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.909442 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:01.940797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.939125 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:01.942119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:01.939287 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:02.059623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:02.059575 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:02.348008 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:16:02.347965 4457 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a0479f_d3fd_4919_9bac_d69caaa62449.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:16:02.409429 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:02.352464 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:16:02.496897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:02.463485 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:02.536594 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:02.536543 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:16:02.561616 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:02.561514 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:16:02.607992 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:02.567481 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:02.828565 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:02.567521 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:02.831423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:02.745300 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:16:02.846298 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:02.745688 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:16:02.912022 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:02.905807 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:03.050782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:02.906358 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:03.108503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.108370 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:16:03.124695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.121737 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:03.153065 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.141532 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:03.153065 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.145420 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:03.169135 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.169081 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:03.197655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.197191 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:03.230901 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:16:03.230856 4457 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.388s" Nov 03 20:16:03.231102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.230947 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:16:03.231102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231067 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:16:03.231224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231111 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:16:03.231224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231144 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:16:03.231224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231181 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:16:03.231224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231218 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:16:03.231451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231263 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:16:03.231451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231293 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:16:03.231451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231330 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:16:03.231451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231361 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:16:03.231451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231448 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:16:03.231718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231483 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:16:03.231718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231516 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:16:03.231718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231547 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:16:03.231718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.231582 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:16:03.240263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.240231 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:16:03.243718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.243693 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:16:03.522967 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.522876 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz"] Nov 03 20:16:03.524728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.524702 4457 cpu_manager.go:414] "containerMap: removing container" podUID="33a0479f-d3fd-4919-9bac-d69caaa62449" containerName="revoke-keys" Nov 03 20:16:03.524728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.524725 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a0479f-d3fd-4919-9bac-d69caaa62449" containerName="revoke-keys" Nov 03 20:16:03.524944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.524870 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a0479f-d3fd-4919-9bac-d69caaa62449" containerName="revoke-keys" Nov 03 20:16:03.533112 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.533080 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" Nov 03 20:16:03.540126 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.540097 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:16:03.540367 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.540337 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:03.543870 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.543820 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz"] Nov 03 20:16:03.650084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.650040 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-012c1ed0-d2mkz\" (UID: \"70996ecc-78bc-49e7-be91-83904b62094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" Nov 03 20:16:03.650267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.650128 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cgp7\" (UniqueName: \"kubernetes.io/projected/70996ecc-78bc-49e7-be91-83904b62094a-kube-api-access-8cgp7\") pod \"maas-api-revoke-sub-012c1ed0-d2mkz\" (UID: \"70996ecc-78bc-49e7-be91-83904b62094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" Nov 03 20:16:03.650345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.650328 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-012c1ed0-d2mkz\" (UID: \"70996ecc-78bc-49e7-be91-83904b62094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" Nov 03 20:16:03.753761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.753719 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-012c1ed0-d2mkz\" (UID: \"70996ecc-78bc-49e7-be91-83904b62094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" Nov 03 20:16:03.753964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.753814 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cgp7\" (UniqueName: \"kubernetes.io/projected/70996ecc-78bc-49e7-be91-83904b62094a-kube-api-access-8cgp7\") pod \"maas-api-revoke-sub-012c1ed0-d2mkz\" (UID: \"70996ecc-78bc-49e7-be91-83904b62094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" Nov 03 20:16:03.754063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.754044 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-012c1ed0-d2mkz\" (UID: \"70996ecc-78bc-49e7-be91-83904b62094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" Nov 03 20:16:03.755685 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.755655 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-012c1ed0-d2mkz\" (UID: \"70996ecc-78bc-49e7-be91-83904b62094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" Nov 03 20:16:03.763137 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.763095 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-012c1ed0-d2mkz\" (UID: \"70996ecc-78bc-49e7-be91-83904b62094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" Nov 03 20:16:03.769413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.769368 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cgp7\" (UniqueName: \"kubernetes.io/projected/70996ecc-78bc-49e7-be91-83904b62094a-kube-api-access-8cgp7\") pod \"maas-api-revoke-sub-012c1ed0-d2mkz\" (UID: \"70996ecc-78bc-49e7-be91-83904b62094a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" Nov 03 20:16:03.908409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:03.899609 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" Nov 03 20:16:04.414822 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:16:04.414788 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70996ecc_78bc_49e7_be91_83904b62094a.slice/crio-75194a17af72f2da57d01122e9711b8c60e5c7106d1cf5f5128f1bfb6f116cff WatchSource:0}: Error finding container 75194a17af72f2da57d01122e9711b8c60e5c7106d1cf5f5128f1bfb6f116cff: Status 404 returned error can't find the container with id 75194a17af72f2da57d01122e9711b8c60e5c7106d1cf5f5128f1bfb6f116cff Nov 03 20:16:04.416924 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:04.416898 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz"] Nov 03 20:16:05.259520 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.259484 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" event={"ID":"70996ecc-78bc-49e7-be91-83904b62094a","Type":"ContainerStarted","Data":"ecdd6639fd6fc016518fba2697ae56319e1b622987a8d0640b0f8777b0cda741"} Nov 03 20:16:05.260024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.259538 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" event={"ID":"70996ecc-78bc-49e7-be91-83904b62094a","Type":"ContainerStarted","Data":"75194a17af72f2da57d01122e9711b8c60e5c7106d1cf5f5128f1bfb6f116cff"} Nov 03 20:16:05.411835 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.411770 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" podStartSLOduration=2.411746824 podStartE2EDuration="2.411746824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:05.339189737 +0000 UTC m=+1143.714786484" watchObservedRunningTime="2026-11-03 20:16:05.411746824 +0000 UTC m=+1143.787343572" Nov 03 20:16:05.441590 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.441367 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd"] Nov 03 20:16:05.461306 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.461269 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd"] Nov 03 20:16:05.461472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.461382 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.475727 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.475602 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 20:16:05.475727 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.475668 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 20:16:05.475727 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.475690 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 20:16:05.544549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.543734 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlk9s\" (UniqueName: \"kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-kube-api-access-hlk9s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.544549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.543849 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.544549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.543972 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.544549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.544096 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.544549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.544160 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.544549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.544207 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.544549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.544243 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.572447 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.571917 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xj9dq\"" Nov 03 20:16:05.645298 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.645267 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.645479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.645441 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.645574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.645482 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.645574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.645527 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.645574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.645563 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.645731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.645589 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.645731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.645675 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlk9s\" (UniqueName: \"kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-kube-api-access-hlk9s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.646624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.646597 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.647881 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.647852 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.647995 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.647938 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.648193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.648173 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.648193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.648188 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.659512 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.659476 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.661224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.661199 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlk9s\" (UniqueName: \"kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-kube-api-access-hlk9s\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lscmd\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:05.872627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:05.872594 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:06.258983 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:16:06.258911 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:16:06.298257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:06.298213 4457 generic.go:356] "Generic (PLEG): container finished" podID="70996ecc-78bc-49e7-be91-83904b62094a" containerID="ecdd6639fd6fc016518fba2697ae56319e1b622987a8d0640b0f8777b0cda741" exitCode=0 Nov 03 20:16:06.298988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:06.298304 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" event={"ID":"70996ecc-78bc-49e7-be91-83904b62094a","Type":"ContainerDied","Data":"ecdd6639fd6fc016518fba2697ae56319e1b622987a8d0640b0f8777b0cda741"} Nov 03 20:16:06.424455 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:16:06.424421 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2965dfe_13cd_4d50_97d1_8d48eb46d9e0.slice/crio-9b92526c24a81b3692050d21af4b5b37be21bd593cc8b764ef6a664750213e36 WatchSource:0}: Error finding container 9b92526c24a81b3692050d21af4b5b37be21bd593cc8b764ef6a664750213e36: Status 404 returned error can't find the container with id 9b92526c24a81b3692050d21af4b5b37be21bd593cc8b764ef6a664750213e36 Nov 03 20:16:06.425751 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:06.425720 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd"] Nov 03 20:16:07.310473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:07.310440 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" event={"ID":"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0","Type":"ContainerStarted","Data":"9b92526c24a81b3692050d21af4b5b37be21bd593cc8b764ef6a664750213e36"} Nov 03 20:16:07.543046 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:07.543011 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" Nov 03 20:16:07.635552 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:07.635462 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70996ecc-78bc-49e7-be91-83904b62094a-kube-api-access-8cgp7\" (UniqueName: \"kubernetes.io/projected/70996ecc-78bc-49e7-be91-83904b62094a-kube-api-access-8cgp7\") pod \"70996ecc-78bc-49e7-be91-83904b62094a\" (UID: \"70996ecc-78bc-49e7-be91-83904b62094a\") " Nov 03 20:16:07.635839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:07.635818 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-xks-ca\") pod \"70996ecc-78bc-49e7-be91-83904b62094a\" (UID: \"70996ecc-78bc-49e7-be91-83904b62094a\") " Nov 03 20:16:07.635902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:07.635886 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-openshift-ca\") pod \"70996ecc-78bc-49e7-be91-83904b62094a\" (UID: \"70996ecc-78bc-49e7-be91-83904b62094a\") " Nov 03 20:16:07.636728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:07.636695 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-openshift-ca" pod "70996ecc-78bc-49e7-be91-83904b62094a" (UID: "70996ecc-78bc-49e7-be91-83904b62094a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:07.638318 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:07.638289 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-xks-ca" pod "70996ecc-78bc-49e7-be91-83904b62094a" (UID: "70996ecc-78bc-49e7-be91-83904b62094a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:07.638460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:07.638301 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70996ecc-78bc-49e7-be91-83904b62094a-kube-api-access-8cgp7" pod "70996ecc-78bc-49e7-be91-83904b62094a" (UID: "70996ecc-78bc-49e7-be91-83904b62094a"). InnerVolumeSpecName "kube-api-access-8cgp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:07.738332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:07.738293 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:07.738332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:07.738330 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70996ecc-78bc-49e7-be91-83904b62094a-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:07.738332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:07.738341 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cgp7\" (UniqueName: \"kubernetes.io/projected/70996ecc-78bc-49e7-be91-83904b62094a-kube-api-access-8cgp7\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:08.323959 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:08.323918 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" event={"ID":"70996ecc-78bc-49e7-be91-83904b62094a","Type":"ContainerDied","Data":"75194a17af72f2da57d01122e9711b8c60e5c7106d1cf5f5128f1bfb6f116cff"} Nov 03 20:16:08.324382 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:08.323986 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75194a17af72f2da57d01122e9711b8c60e5c7106d1cf5f5128f1bfb6f116cff" Nov 03 20:16:08.324382 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:08.323938 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz" Nov 03 20:16:09.323257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.323220 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq"] Nov 03 20:16:09.324361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.324342 4457 cpu_manager.go:414] "containerMap: removing container" podUID="70996ecc-78bc-49e7-be91-83904b62094a" containerName="revoke-keys" Nov 03 20:16:09.324361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.324363 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70996ecc-78bc-49e7-be91-83904b62094a" containerName="revoke-keys" Nov 03 20:16:09.324811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.324554 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="70996ecc-78bc-49e7-be91-83904b62094a" containerName="revoke-keys" Nov 03 20:16:09.330230 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.330207 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" Nov 03 20:16:09.334375 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.334344 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq"] Nov 03 20:16:09.347956 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.347920 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:09.347956 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.347949 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:16:09.380306 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.379361 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-301728c8-7vksq\" (UID: \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" Nov 03 20:16:09.380306 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.379458 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhxwv\" (UniqueName: \"kubernetes.io/projected/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-kube-api-access-fhxwv\") pod \"maas-api-revoke-sub-301728c8-7vksq\" (UID: \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" Nov 03 20:16:09.380306 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.379876 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-301728c8-7vksq\" (UID: \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" Nov 03 20:16:09.481233 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.481194 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-301728c8-7vksq\" (UID: \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" Nov 03 20:16:09.481488 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.481264 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-301728c8-7vksq\" (UID: \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" Nov 03 20:16:09.481488 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.481307 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhxwv\" (UniqueName: \"kubernetes.io/projected/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-kube-api-access-fhxwv\") pod \"maas-api-revoke-sub-301728c8-7vksq\" (UID: \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" Nov 03 20:16:09.482087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.482059 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-301728c8-7vksq\" (UID: \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" Nov 03 20:16:09.485170 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.485140 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-301728c8-7vksq\" (UID: \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" Nov 03 20:16:09.496847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.496818 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhxwv\" (UniqueName: \"kubernetes.io/projected/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-kube-api-access-fhxwv\") pod \"maas-api-revoke-sub-301728c8-7vksq\" (UID: \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" Nov 03 20:16:09.680530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:09.680431 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" Nov 03 20:16:10.013210 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:16:10.013174 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6551b5_3042_42e0_a09c_e8ec9b50be1f.slice/crio-9d6109b1341b1146f47bc86b0096ac6d186dd6234873ee681785fd346d15cdff WatchSource:0}: Error finding container 9d6109b1341b1146f47bc86b0096ac6d186dd6234873ee681785fd346d15cdff: Status 404 returned error can't find the container with id 9d6109b1341b1146f47bc86b0096ac6d186dd6234873ee681785fd346d15cdff Nov 03 20:16:10.014381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:10.014354 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq"] Nov 03 20:16:10.371697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:10.371597 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" event={"ID":"0e6551b5-3042-42e0-a09c-e8ec9b50be1f","Type":"ContainerStarted","Data":"9d6109b1341b1146f47bc86b0096ac6d186dd6234873ee681785fd346d15cdff"} Nov 03 20:16:10.812130 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:10.812097 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:10.812217 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:10.812175 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:16:11.402468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:11.394239 4457 generic.go:356] "Generic (PLEG): container finished" podID="0e6551b5-3042-42e0-a09c-e8ec9b50be1f" containerID="46dce2e31628953bfe1a9bb7d330a5b0ec2e177aafb8c135a4da4d0fc39f1e72" exitCode=0 Nov 03 20:16:11.402468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:11.394380 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" event={"ID":"0e6551b5-3042-42e0-a09c-e8ec9b50be1f","Type":"ContainerDied","Data":"46dce2e31628953bfe1a9bb7d330a5b0ec2e177aafb8c135a4da4d0fc39f1e72"} Nov 03 20:16:11.402468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:11.397079 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" event={"ID":"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0","Type":"ContainerStarted","Data":"fb9ba3ad9cb48b15d2748c2a05914463e7b05f1df6744ffaab9aa0c99f8ab6ad"} Nov 03 20:16:11.402468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:11.397241 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:11.402468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:11.400145 4457 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lscmd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:16:11.402468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:11.400188 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" podUID="f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:16:12.407533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:12.407379 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:16:12.436637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:12.436579 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" podStartSLOduration=3.053111615 podStartE2EDuration="7.436564204s" totalImagesPullingTime="4.383452589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:11.453674777 +0000 UTC m=+1149.829271525" watchObservedRunningTime="2026-11-03 20:16:12.436564204 +0000 UTC m=+1150.812160950" Nov 03 20:16:12.650077 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:12.650050 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" Nov 03 20:16:12.666895 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:12.666819 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-xks-ca\") pod \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\" (UID: \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\") " Nov 03 20:16:12.667036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:12.666904 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-kube-api-access-fhxwv\" (UniqueName: \"kubernetes.io/projected/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-kube-api-access-fhxwv\") pod \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\" (UID: \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\") " Nov 03 20:16:12.667173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:12.667153 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-openshift-ca\") pod \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\" (UID: \"0e6551b5-3042-42e0-a09c-e8ec9b50be1f\") " Nov 03 20:16:12.667614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:12.667589 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-openshift-ca" pod "0e6551b5-3042-42e0-a09c-e8ec9b50be1f" (UID: "0e6551b5-3042-42e0-a09c-e8ec9b50be1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:12.667889 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:12.667870 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:12.669549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:12.669522 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-kube-api-access-fhxwv" pod "0e6551b5-3042-42e0-a09c-e8ec9b50be1f" (UID: "0e6551b5-3042-42e0-a09c-e8ec9b50be1f"). InnerVolumeSpecName "kube-api-access-fhxwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:12.669802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:12.669714 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-xks-ca" pod "0e6551b5-3042-42e0-a09c-e8ec9b50be1f" (UID: "0e6551b5-3042-42e0-a09c-e8ec9b50be1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:12.769818 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:12.769766 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:12.769818 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:12.769816 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhxwv\" (UniqueName: \"kubernetes.io/projected/0e6551b5-3042-42e0-a09c-e8ec9b50be1f-kube-api-access-fhxwv\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:13.417274 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:13.417239 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" Nov 03 20:16:13.417803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:13.417267 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq" event={"ID":"0e6551b5-3042-42e0-a09c-e8ec9b50be1f","Type":"ContainerDied","Data":"9d6109b1341b1146f47bc86b0096ac6d186dd6234873ee681785fd346d15cdff"} Nov 03 20:16:13.417803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:13.417323 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d6109b1341b1146f47bc86b0096ac6d186dd6234873ee681785fd346d15cdff" Nov 03 20:16:15.732991 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.732941 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl"] Nov 03 20:16:15.733723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.733701 4457 cpu_manager.go:414] "containerMap: removing container" podUID="0e6551b5-3042-42e0-a09c-e8ec9b50be1f" containerName="revoke-keys" Nov 03 20:16:15.733723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.733721 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6551b5-3042-42e0-a09c-e8ec9b50be1f" containerName="revoke-keys" Nov 03 20:16:15.733943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.733847 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e6551b5-3042-42e0-a09c-e8ec9b50be1f" containerName="revoke-keys" Nov 03 20:16:15.737981 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.737960 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" Nov 03 20:16:15.765428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.746924 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:15.765428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.747054 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:16:15.879596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.879366 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c847b846-5g5pl\" (UID: \"c8300397-1ed0-4692-9a45-ebed514727fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" Nov 03 20:16:15.879596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.879473 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c847b846-5g5pl\" (UID: \"c8300397-1ed0-4692-9a45-ebed514727fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" Nov 03 20:16:15.880006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.879546 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcjhm\" (UniqueName: \"kubernetes.io/projected/c8300397-1ed0-4692-9a45-ebed514727fc-kube-api-access-rcjhm\") pod \"maas-api-revoke-keys-e2e-ait-c847b846-5g5pl\" (UID: \"c8300397-1ed0-4692-9a45-ebed514727fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" Nov 03 20:16:15.993504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.993051 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c847b846-5g5pl\" (UID: \"c8300397-1ed0-4692-9a45-ebed514727fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" Nov 03 20:16:15.993504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.993142 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c847b846-5g5pl\" (UID: \"c8300397-1ed0-4692-9a45-ebed514727fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" Nov 03 20:16:15.993504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.993197 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcjhm\" (UniqueName: \"kubernetes.io/projected/c8300397-1ed0-4692-9a45-ebed514727fc-kube-api-access-rcjhm\") pod \"maas-api-revoke-keys-e2e-ait-c847b846-5g5pl\" (UID: \"c8300397-1ed0-4692-9a45-ebed514727fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" Nov 03 20:16:16.001455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:15.996067 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-c847b846-5g5pl\" (UID: \"c8300397-1ed0-4692-9a45-ebed514727fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" Nov 03 20:16:16.023151 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:16.022958 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcjhm\" (UniqueName: \"kubernetes.io/projected/c8300397-1ed0-4692-9a45-ebed514727fc-kube-api-access-rcjhm\") pod \"maas-api-revoke-keys-e2e-ait-c847b846-5g5pl\" (UID: \"c8300397-1ed0-4692-9a45-ebed514727fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" Nov 03 20:16:16.023776 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:16.023751 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-c847b846-5g5pl\" (UID: \"c8300397-1ed0-4692-9a45-ebed514727fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" Nov 03 20:16:16.031163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:16.031135 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl"] Nov 03 20:16:16.311614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:16.311580 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" Nov 03 20:16:16.603921 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:16.603844 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl"] Nov 03 20:16:16.609929 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:16:16.609899 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8300397_1ed0_4692_9a45_ebed514727fc.slice/crio-cecbeb8b5c5774bf0ebd0345b9eb5abe39c519cdd7d94b323ead1eef00750e95 WatchSource:0}: Error finding container cecbeb8b5c5774bf0ebd0345b9eb5abe39c519cdd7d94b323ead1eef00750e95: Status 404 returned error can't find the container with id cecbeb8b5c5774bf0ebd0345b9eb5abe39c519cdd7d94b323ead1eef00750e95 Nov 03 20:16:17.467724 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:17.467675 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" event={"ID":"c8300397-1ed0-4692-9a45-ebed514727fc","Type":"ContainerStarted","Data":"ce1d8b7d61bf2a99c1db87f1402ae09b18417fecc172d7aaed043f9e51aee184"} Nov 03 20:16:17.468253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:17.467840 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" event={"ID":"c8300397-1ed0-4692-9a45-ebed514727fc","Type":"ContainerStarted","Data":"cecbeb8b5c5774bf0ebd0345b9eb5abe39c519cdd7d94b323ead1eef00750e95"} Nov 03 20:16:17.510676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:17.510615 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" podStartSLOduration=2.510598542 podStartE2EDuration="2.510598542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:16:17.509298366 +0000 UTC m=+1155.884895114" watchObservedRunningTime="2026-11-03 20:16:17.510598542 +0000 UTC m=+1155.886195289" Nov 03 20:16:18.498938 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:18.498898 4457 generic.go:356] "Generic (PLEG): container finished" podID="c8300397-1ed0-4692-9a45-ebed514727fc" containerID="ce1d8b7d61bf2a99c1db87f1402ae09b18417fecc172d7aaed043f9e51aee184" exitCode=0 Nov 03 20:16:18.499763 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:18.499364 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" event={"ID":"c8300397-1ed0-4692-9a45-ebed514727fc","Type":"ContainerDied","Data":"ce1d8b7d61bf2a99c1db87f1402ae09b18417fecc172d7aaed043f9e51aee184"} Nov 03 20:16:19.699474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:19.699446 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" Nov 03 20:16:19.868463 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:19.868431 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-xks-ca\") pod \"c8300397-1ed0-4692-9a45-ebed514727fc\" (UID: \"c8300397-1ed0-4692-9a45-ebed514727fc\") " Nov 03 20:16:19.868688 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:19.868665 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8300397-1ed0-4692-9a45-ebed514727fc-kube-api-access-rcjhm\" (UniqueName: \"kubernetes.io/projected/c8300397-1ed0-4692-9a45-ebed514727fc-kube-api-access-rcjhm\") pod \"c8300397-1ed0-4692-9a45-ebed514727fc\" (UID: \"c8300397-1ed0-4692-9a45-ebed514727fc\") " Nov 03 20:16:19.868780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:19.868724 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-openshift-ca\") pod \"c8300397-1ed0-4692-9a45-ebed514727fc\" (UID: \"c8300397-1ed0-4692-9a45-ebed514727fc\") " Nov 03 20:16:19.869434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:19.869365 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-openshift-ca" pod "c8300397-1ed0-4692-9a45-ebed514727fc" (UID: "c8300397-1ed0-4692-9a45-ebed514727fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:19.869734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:19.869710 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:19.871140 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:19.871113 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-xks-ca" pod "c8300397-1ed0-4692-9a45-ebed514727fc" (UID: "c8300397-1ed0-4692-9a45-ebed514727fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:19.871323 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:19.871305 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8300397-1ed0-4692-9a45-ebed514727fc-kube-api-access-rcjhm" pod "c8300397-1ed0-4692-9a45-ebed514727fc" (UID: "c8300397-1ed0-4692-9a45-ebed514727fc"). InnerVolumeSpecName "kube-api-access-rcjhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:19.972063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:19.971702 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8300397-1ed0-4692-9a45-ebed514727fc-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:19.972063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:19.971748 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcjhm\" (UniqueName: \"kubernetes.io/projected/c8300397-1ed0-4692-9a45-ebed514727fc-kube-api-access-rcjhm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:20.525889 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.525854 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" Nov 03 20:16:20.526070 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.525884 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl" event={"ID":"c8300397-1ed0-4692-9a45-ebed514727fc","Type":"ContainerDied","Data":"cecbeb8b5c5774bf0ebd0345b9eb5abe39c519cdd7d94b323ead1eef00750e95"} Nov 03 20:16:20.526070 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.525937 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cecbeb8b5c5774bf0ebd0345b9eb5abe39c519cdd7d94b323ead1eef00750e95" Nov 03 20:16:20.746693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.746657 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf"] Nov 03 20:16:20.747403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.747374 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c8300397-1ed0-4692-9a45-ebed514727fc" containerName="revoke-keys" Nov 03 20:16:20.747403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.747403 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8300397-1ed0-4692-9a45-ebed514727fc" containerName="revoke-keys" Nov 03 20:16:20.747548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.747538 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8300397-1ed0-4692-9a45-ebed514727fc" containerName="revoke-keys" Nov 03 20:16:20.751554 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.751533 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" Nov 03 20:16:20.757845 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.757815 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:16:20.758161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.758140 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:16:20.761151 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.761123 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf"] Nov 03 20:16:20.887798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.887723 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz9sf\" (UniqueName: \"kubernetes.io/projected/b31d8a65-79af-484c-8628-b5fcf2e26158-kube-api-access-wz9sf\") pod \"maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf\" (UID: \"b31d8a65-79af-484c-8628-b5fcf2e26158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" Nov 03 20:16:20.887981 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.887828 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf\" (UID: \"b31d8a65-79af-484c-8628-b5fcf2e26158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" Nov 03 20:16:20.887981 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.887944 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf\" (UID: \"b31d8a65-79af-484c-8628-b5fcf2e26158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" Nov 03 20:16:20.989438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.989366 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf\" (UID: \"b31d8a65-79af-484c-8628-b5fcf2e26158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" Nov 03 20:16:20.989606 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.989561 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf\" (UID: \"b31d8a65-79af-484c-8628-b5fcf2e26158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" Nov 03 20:16:20.989775 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.989760 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz9sf\" (UniqueName: \"kubernetes.io/projected/b31d8a65-79af-484c-8628-b5fcf2e26158-kube-api-access-wz9sf\") pod \"maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf\" (UID: \"b31d8a65-79af-484c-8628-b5fcf2e26158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" Nov 03 20:16:20.990232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.990204 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf\" (UID: \"b31d8a65-79af-484c-8628-b5fcf2e26158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" Nov 03 20:16:20.992128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:20.992100 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf\" (UID: \"b31d8a65-79af-484c-8628-b5fcf2e26158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" Nov 03 20:16:21.006330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:21.006298 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz9sf\" (UniqueName: \"kubernetes.io/projected/b31d8a65-79af-484c-8628-b5fcf2e26158-kube-api-access-wz9sf\") pod \"maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf\" (UID: \"b31d8a65-79af-484c-8628-b5fcf2e26158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" Nov 03 20:16:21.074624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:21.074591 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" Nov 03 20:16:21.407955 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:16:21.407913 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31d8a65_79af_484c_8628_b5fcf2e26158.slice/crio-e433d317605cf0744d08b3b2a778d0eef5df7f3703714b3f683d4353c2276374 WatchSource:0}: Error finding container e433d317605cf0744d08b3b2a778d0eef5df7f3703714b3f683d4353c2276374: Status 404 returned error can't find the container with id e433d317605cf0744d08b3b2a778d0eef5df7f3703714b3f683d4353c2276374 Nov 03 20:16:21.408648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:21.408565 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf"] Nov 03 20:16:21.536560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:21.536522 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" event={"ID":"b31d8a65-79af-484c-8628-b5fcf2e26158","Type":"ContainerStarted","Data":"e433d317605cf0744d08b3b2a778d0eef5df7f3703714b3f683d4353c2276374"} Nov 03 20:16:22.429719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:22.429690 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p"] Nov 03 20:16:22.445569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:22.445532 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj"] Nov 03 20:16:22.551208 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:22.551167 4457 generic.go:356] "Generic (PLEG): container finished" podID="b31d8a65-79af-484c-8628-b5fcf2e26158" containerID="0efbd5099d4517aa88482212252a70178fe46c5312f25c4fcfd7403d59162fc6" exitCode=0 Nov 03 20:16:22.551444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:22.551244 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" event={"ID":"b31d8a65-79af-484c-8628-b5fcf2e26158","Type":"ContainerDied","Data":"0efbd5099d4517aa88482212252a70178fe46c5312f25c4fcfd7403d59162fc6"} Nov 03 20:16:23.754435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:23.754402 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" Nov 03 20:16:23.851986 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:23.851874 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-openshift-ca\") pod \"b31d8a65-79af-484c-8628-b5fcf2e26158\" (UID: \"b31d8a65-79af-484c-8628-b5fcf2e26158\") " Nov 03 20:16:23.852166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:23.852148 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-xks-ca\") pod \"b31d8a65-79af-484c-8628-b5fcf2e26158\" (UID: \"b31d8a65-79af-484c-8628-b5fcf2e26158\") " Nov 03 20:16:23.852311 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:23.852291 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31d8a65-79af-484c-8628-b5fcf2e26158-kube-api-access-wz9sf\" (UniqueName: \"kubernetes.io/projected/b31d8a65-79af-484c-8628-b5fcf2e26158-kube-api-access-wz9sf\") pod \"b31d8a65-79af-484c-8628-b5fcf2e26158\" (UID: \"b31d8a65-79af-484c-8628-b5fcf2e26158\") " Nov 03 20:16:23.852628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:23.852600 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-openshift-ca" pod "b31d8a65-79af-484c-8628-b5fcf2e26158" (UID: "b31d8a65-79af-484c-8628-b5fcf2e26158"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:23.853151 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:23.853126 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:23.854908 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:23.854881 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-xks-ca" pod "b31d8a65-79af-484c-8628-b5fcf2e26158" (UID: "b31d8a65-79af-484c-8628-b5fcf2e26158"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:23.855195 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:23.855158 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31d8a65-79af-484c-8628-b5fcf2e26158-kube-api-access-wz9sf" pod "b31d8a65-79af-484c-8628-b5fcf2e26158" (UID: "b31d8a65-79af-484c-8628-b5fcf2e26158"). InnerVolumeSpecName "kube-api-access-wz9sf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:23.954266 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:23.954234 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31d8a65-79af-484c-8628-b5fcf2e26158-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:23.954266 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:23.954265 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz9sf\" (UniqueName: \"kubernetes.io/projected/b31d8a65-79af-484c-8628-b5fcf2e26158-kube-api-access-wz9sf\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:24.575053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:24.575024 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" Nov 03 20:16:24.575236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:24.575020 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf" event={"ID":"b31d8a65-79af-484c-8628-b5fcf2e26158","Type":"ContainerDied","Data":"e433d317605cf0744d08b3b2a778d0eef5df7f3703714b3f683d4353c2276374"} Nov 03 20:16:24.575236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:24.575156 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e433d317605cf0744d08b3b2a778d0eef5df7f3703714b3f683d4353c2276374" Nov 03 20:16:27.430996 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:27.430863 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" podUID="79caffd6-766b-4a72-951d-901ee5497282" containerName="payload-processing" containerID="cri-o://634fbca07ae8e899ab5019bb147eb57b1395b993ee835de8255a9abd13419aa8" gracePeriod=25 Nov 03 20:16:27.448554 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:27.445890 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" podUID="9d3f423c-4451-4f01-8be1-44bf84c13e9a" containerName="payload-pre-processing" containerID="cri-o://5c0bf30ee9c2030f1ef350061b9890ff9c4456b2122f1a81d44ac02bc9dbf5af" gracePeriod=25 Nov 03 20:16:27.703433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:27.699959 4457 generic.go:356] "Generic (PLEG): container finished" podID="79caffd6-766b-4a72-951d-901ee5497282" containerID="634fbca07ae8e899ab5019bb147eb57b1395b993ee835de8255a9abd13419aa8" exitCode=0 Nov 03 20:16:27.703433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:27.700210 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" event={"ID":"79caffd6-766b-4a72-951d-901ee5497282","Type":"ContainerDied","Data":"634fbca07ae8e899ab5019bb147eb57b1395b993ee835de8255a9abd13419aa8"} Nov 03 20:16:28.356876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.356845 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" Nov 03 20:16:28.359355 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.359322 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79caffd6-766b-4a72-951d-901ee5497282-kube-api-access-ljndd\" (UniqueName: \"kubernetes.io/projected/79caffd6-766b-4a72-951d-901ee5497282-kube-api-access-ljndd\") pod \"79caffd6-766b-4a72-951d-901ee5497282\" (UID: \"79caffd6-766b-4a72-951d-901ee5497282\") " Nov 03 20:16:28.359594 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.359412 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79caffd6-766b-4a72-951d-901ee5497282-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79caffd6-766b-4a72-951d-901ee5497282-plugins-config-volume\") pod \"79caffd6-766b-4a72-951d-901ee5497282\" (UID: \"79caffd6-766b-4a72-951d-901ee5497282\") " Nov 03 20:16:28.360489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.360460 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79caffd6-766b-4a72-951d-901ee5497282-plugins-config-volume" pod "79caffd6-766b-4a72-951d-901ee5497282" (UID: "79caffd6-766b-4a72-951d-901ee5497282"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:28.362557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.362524 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79caffd6-766b-4a72-951d-901ee5497282-kube-api-access-ljndd" pod "79caffd6-766b-4a72-951d-901ee5497282" (UID: "79caffd6-766b-4a72-951d-901ee5497282"). InnerVolumeSpecName "kube-api-access-ljndd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:28.363654 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.363626 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" Nov 03 20:16:28.461062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.460828 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d3f423c-4451-4f01-8be1-44bf84c13e9a-kube-api-access-gj9bq\" (UniqueName: \"kubernetes.io/projected/9d3f423c-4451-4f01-8be1-44bf84c13e9a-kube-api-access-gj9bq\") pod \"9d3f423c-4451-4f01-8be1-44bf84c13e9a\" (UID: \"9d3f423c-4451-4f01-8be1-44bf84c13e9a\") " Nov 03 20:16:28.461702 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.461678 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d3f423c-4451-4f01-8be1-44bf84c13e9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d3f423c-4451-4f01-8be1-44bf84c13e9a-plugins-config-volume\") pod \"9d3f423c-4451-4f01-8be1-44bf84c13e9a\" (UID: \"9d3f423c-4451-4f01-8be1-44bf84c13e9a\") " Nov 03 20:16:28.462201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.462169 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d3f423c-4451-4f01-8be1-44bf84c13e9a-plugins-config-volume" pod "9d3f423c-4451-4f01-8be1-44bf84c13e9a" (UID: "9d3f423c-4451-4f01-8be1-44bf84c13e9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:28.462851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.462590 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljndd\" (UniqueName: \"kubernetes.io/projected/79caffd6-766b-4a72-951d-901ee5497282-kube-api-access-ljndd\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:28.462851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.462615 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79caffd6-766b-4a72-951d-901ee5497282-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:28.462851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.462633 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d3f423c-4451-4f01-8be1-44bf84c13e9a-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:28.463869 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.463825 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d3f423c-4451-4f01-8be1-44bf84c13e9a-kube-api-access-gj9bq" pod "9d3f423c-4451-4f01-8be1-44bf84c13e9a" (UID: "9d3f423c-4451-4f01-8be1-44bf84c13e9a"). InnerVolumeSpecName "kube-api-access-gj9bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:28.569436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.567409 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj9bq\" (UniqueName: \"kubernetes.io/projected/9d3f423c-4451-4f01-8be1-44bf84c13e9a-kube-api-access-gj9bq\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:28.747418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.745286 4457 generic.go:356] "Generic (PLEG): container finished" podID="9d3f423c-4451-4f01-8be1-44bf84c13e9a" containerID="5c0bf30ee9c2030f1ef350061b9890ff9c4456b2122f1a81d44ac02bc9dbf5af" exitCode=0 Nov 03 20:16:28.747418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.745578 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" event={"ID":"9d3f423c-4451-4f01-8be1-44bf84c13e9a","Type":"ContainerDied","Data":"5c0bf30ee9c2030f1ef350061b9890ff9c4456b2122f1a81d44ac02bc9dbf5af"} Nov 03 20:16:28.747418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.745651 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" event={"ID":"9d3f423c-4451-4f01-8be1-44bf84c13e9a","Type":"ContainerDied","Data":"029f3d9d107598e10adad411a9d592c186c397930b9432db50973266b4aa9467"} Nov 03 20:16:28.747418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.745686 4457 scope.go:122] "RemoveContainer" containerID="5c0bf30ee9c2030f1ef350061b9890ff9c4456b2122f1a81d44ac02bc9dbf5af" Nov 03 20:16:28.747418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.745916 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj" Nov 03 20:16:28.818453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.801846 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" event={"ID":"79caffd6-766b-4a72-951d-901ee5497282","Type":"ContainerDied","Data":"1a2867f5426579ab58843bf2fc0eb2da793c0f7f25404d102c5a9b2b5b558465"} Nov 03 20:16:28.818453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.801994 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p" Nov 03 20:16:28.878183 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.878144 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p"] Nov 03 20:16:28.879697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.879669 4457 scope.go:122] "RemoveContainer" containerID="5c0bf30ee9c2030f1ef350061b9890ff9c4456b2122f1a81d44ac02bc9dbf5af" Nov 03 20:16:28.880124 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:16:28.880090 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0bf30ee9c2030f1ef350061b9890ff9c4456b2122f1a81d44ac02bc9dbf5af\": container with ID starting with 5c0bf30ee9c2030f1ef350061b9890ff9c4456b2122f1a81d44ac02bc9dbf5af not found: ID does not exist" containerID="5c0bf30ee9c2030f1ef350061b9890ff9c4456b2122f1a81d44ac02bc9dbf5af" Nov 03 20:16:28.880224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.880138 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0bf30ee9c2030f1ef350061b9890ff9c4456b2122f1a81d44ac02bc9dbf5af"} err="failed to get container status \"5c0bf30ee9c2030f1ef350061b9890ff9c4456b2122f1a81d44ac02bc9dbf5af\": rpc error: code = NotFound desc = could not find container \"5c0bf30ee9c2030f1ef350061b9890ff9c4456b2122f1a81d44ac02bc9dbf5af\": container with ID starting with 5c0bf30ee9c2030f1ef350061b9890ff9c4456b2122f1a81d44ac02bc9dbf5af not found: ID does not exist" Nov 03 20:16:28.880224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.880162 4457 scope.go:122] "RemoveContainer" containerID="634fbca07ae8e899ab5019bb147eb57b1395b993ee835de8255a9abd13419aa8" Nov 03 20:16:28.901409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.896498 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0416e8f1-686f959f96-znh2p"] Nov 03 20:16:28.928317 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.928175 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj"] Nov 03 20:16:28.951442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:28.947331 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0416e8f1-5d844bdb99-nzmcj"] Nov 03 20:16:29.010472 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:16:29.009232 4457 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79caffd6_766b_4a72_951d_901ee5497282.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3f423c_4451_4f01_8be1_44bf84c13e9a.slice/crio-029f3d9d107598e10adad411a9d592c186c397930b9432db50973266b4aa9467\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d3f423c_4451_4f01_8be1_44bf84c13e9a.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:16:29.799409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:29.799199 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79caffd6-766b-4a72-951d-901ee5497282" path="/var/lib/kubelet/pods/79caffd6-766b-4a72-951d-901ee5497282/volumes" Nov 03 20:16:29.816474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:29.815625 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d3f423c-4451-4f01-8be1-44bf84c13e9a" path="/var/lib/kubelet/pods/9d3f423c-4451-4f01-8be1-44bf84c13e9a/volumes" Nov 03 20:16:30.876737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:30.876693 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz"] Nov 03 20:16:30.877630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:30.877591 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" podUID="8a708372-3610-4b08-b671-c76e6ef46675" containerName="maas-api" containerID="cri-o://eaecee9ddadec2e2e4625987fd363466fc40a2cb2be2aaa9bf37a62ff8ed0492" gracePeriod=30 Nov 03 20:16:31.036411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:31.019697 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:31.193061 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:31.192945 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:31.292898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:31.292794 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:31.389543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:31.389489 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:31.548500 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:31.547656 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:31.943473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:31.942489 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.038439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.026858 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.038439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.027951 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:16:32.115046 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.101826 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.243645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.243526 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.246596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.246546 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.271578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.271523 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.273173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.272435 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:16:32.294362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.293653 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:16:32.303570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.303520 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:16:32.401788 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.401744 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.403148 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.403120 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:16:32.576363 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.557590 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.576363 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.558599 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:16:32.646838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.625758 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:32.769018 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.768753 4457 generic.go:356] "Generic (PLEG): container finished" podID="8a708372-3610-4b08-b671-c76e6ef46675" containerID="eaecee9ddadec2e2e4625987fd363466fc40a2cb2be2aaa9bf37a62ff8ed0492" exitCode=0 Nov 03 20:16:32.769018 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.768852 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" event={"ID":"8a708372-3610-4b08-b671-c76e6ef46675","Type":"ContainerDied","Data":"eaecee9ddadec2e2e4625987fd363466fc40a2cb2be2aaa9bf37a62ff8ed0492"} Nov 03 20:16:32.855711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.855597 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:16:32.897653 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.896878 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:33.014414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:32.955300 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:33.112436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.096266 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:33.751232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.725760 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:33.751601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.136954 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:16:33.752547 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.752496 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:33.821052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.170595 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:16:33.905330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.327208 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:33.964845 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.964524 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:33.967701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.327288 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:33.968645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.328762 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:16:34.034854 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.418016 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:34.043203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.674797 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:34.045537 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.752770 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:16:34.046040 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:34.046015 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:16:34.047772 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.151460 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": context deadline exceeded" Nov 03 20:16:34.048997 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.764088 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:16:34.059034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.764316 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:16:34.088772 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.327267 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:34.090302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.980547 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:34.116833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:34.056545 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:16:34.116833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.890369 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:34.140972 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:33.151424 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:34.255473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:34.159121 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:16:34.404602 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:34.404541 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:34.742714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:34.731327 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:16:34.902805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:34.902741 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:34.910154 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:34.903127 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:16:34.914234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:34.914200 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:16:35.013926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:34.954249 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:16:35.014245 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:34.981882 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:16:35.014460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.014440 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:16:35.014590 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.014576 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl"] Nov 03 20:16:35.014717 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.014705 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:16:35.014820 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.014808 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-c847b846-5g5pl"] Nov 03 20:16:35.015029 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.015016 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:16:35.015155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.015144 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:16:35.020505 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.020469 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.030865 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:34.981914 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:16:35.030865 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.029854 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:16:35.100241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.100190 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.113214 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.101794 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:16:35.114972 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.101868 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.116052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.104109 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a708372-3610-4b08-b671-c76e6ef46675-kube-api-access-rnwnl\" (UniqueName: \"kubernetes.io/projected/8a708372-3610-4b08-b671-c76e6ef46675-kube-api-access-rnwnl\") pod \"8a708372-3610-4b08-b671-c76e6ef46675\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " Nov 03 20:16:35.116312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.116294 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-maas-api-tls\") pod \"8a708372-3610-4b08-b671-c76e6ef46675\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " Nov 03 20:16:35.116704 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.116682 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-metrics-certs\") pod \"8a708372-3610-4b08-b671-c76e6ef46675\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " Nov 03 20:16:35.116856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.116841 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a708372-3610-4b08-b671-c76e6ef46675-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a708372-3610-4b08-b671-c76e6ef46675-openshift-service-ca\") pod \"8a708372-3610-4b08-b671-c76e6ef46675\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " Nov 03 20:16:35.118801 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.115225 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.125886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.115326 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.127603 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.115268 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:16:35.174878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.115288 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:16:35.176733 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.176701 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a708372-3610-4b08-b671-c76e6ef46675-openshift-service-ca" pod "8a708372-3610-4b08-b671-c76e6ef46675" (UID: "8a708372-3610-4b08-b671-c76e6ef46675"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:35.176973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.176957 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:16:35.221003 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.220962 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Nov 03 20:16:35.259435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.230509 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.259435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.232729 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.259435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.245249 4457 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a708372-3610-4b08-b671-c76e6ef46675-openshift-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:35.300998 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.261337 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded" Nov 03 20:16:35.300998 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.268943 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.554528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.554436 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-maas-api-tls" pod "8a708372-3610-4b08-b671-c76e6ef46675" (UID: "8a708372-3610-4b08-b671-c76e6ef46675"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:35.562866 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.562831 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-maas-api-tls\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:35.566997 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.566967 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:16:35.635336 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.633230 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:16:35.635336 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.633299 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:16:35.635336 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.633340 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:16:35.635336 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.633375 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:16:35.760349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.760293 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.845415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.843342 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-metrics-certs" pod "8a708372-3610-4b08-b671-c76e6ef46675" (UID: "8a708372-3610-4b08-b671-c76e6ef46675"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:16:35.916455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.914062 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a708372-3610-4b08-b671-c76e6ef46675-kube-api-access-rnwnl" pod "8a708372-3610-4b08-b671-c76e6ef46675" (UID: "8a708372-3610-4b08-b671-c76e6ef46675"). InnerVolumeSpecName "kube-api-access-rnwnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:35.924870 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.924825 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:35.939413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.927522 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a708372-3610-4b08-b671-c76e6ef46675-kube-api-access-rnwnl\" (UniqueName: \"kubernetes.io/projected/8a708372-3610-4b08-b671-c76e6ef46675-kube-api-access-rnwnl\") pod \"8a708372-3610-4b08-b671-c76e6ef46675\" (UID: \"8a708372-3610-4b08-b671-c76e6ef46675\") " Nov 03 20:16:35.944725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.944681 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8300397-1ed0-4692-9a45-ebed514727fc" path="/var/lib/kubelet/pods/c8300397-1ed0-4692-9a45-ebed514727fc/volumes" Nov 03 20:16:35.961988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.960465 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:16:35.962266 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.962246 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:16:35.962362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:35.949892 4457 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a708372-3610-4b08-b671-c76e6ef46675-metrics-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:35.981708 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:16:35.949957 4457 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/8a708372-3610-4b08-b671-c76e6ef46675/volumes/kubernetes.io~projected/kube-api-access-rnwnl Nov 03 20:16:36.048127 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.048089 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a708372-3610-4b08-b671-c76e6ef46675-kube-api-access-rnwnl" pod "8a708372-3610-4b08-b671-c76e6ef46675" (UID: "8a708372-3610-4b08-b671-c76e6ef46675"). InnerVolumeSpecName "kube-api-access-rnwnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:36.049128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.036661 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:36.052182 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.026931 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:36.093471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.080315 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnwnl\" (UniqueName: \"kubernetes.io/projected/8a708372-3610-4b08-b671-c76e6ef46675-kube-api-access-rnwnl\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:36.125629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.125564 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:36.135672 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.135620 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:36.165444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.148840 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:16:36.177126 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.177091 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:16:36.230455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.228311 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" podUID="8a708372-3610-4b08-b671-c76e6ef46675" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.121:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:36.234676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.234627 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:36.400642 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.400543 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:16:36.505415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.500641 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:16:36.546466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.545646 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:16:36.552452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.547984 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" Nov 03 20:16:36.600413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.561027 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz" event={"ID":"8a708372-3610-4b08-b671-c76e6ef46675","Type":"ContainerDied","Data":"af40a0c6c9aeb63157d059ed3f4b2afef4453d17b3e5341c1dfa34906384a244"} Nov 03 20:16:36.600413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.561102 4457 scope.go:122] "RemoveContainer" containerID="eaecee9ddadec2e2e4625987fd363466fc40a2cb2be2aaa9bf37a62ff8ed0492" Nov 03 20:16:36.624416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.602774 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:16:36.841429 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:36.764250 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:37.150307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:37.150257 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:16:37.193412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:37.172706 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:37.193412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:37.172981 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:16:37.434485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:37.298343 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 03 20:16:37.653187 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:37.653139 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:16:37.773246 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:16:37.773126 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:16:38.030627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.030513 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.287460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.042023 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.287460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.152496 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:16:38.287460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.155146 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 03 20:16:38.287460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.165595 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.287460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.180933 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.337412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.185802 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-5bz7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:38.374414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.344586 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.374414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.339864 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.374414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.233578 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.374414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.210745 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-dgjgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:16:38.374414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.370559 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-dgjgp" podUID="8f7c5959-df8b-4348-8c8c-c61ba06f66d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.413399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.384634 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:16:38.569412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.556057 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.861544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.815783 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": context deadline exceeded" Nov 03 20:16:38.861544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.816116 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:16:38.942000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.941953 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:38.964792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:38.964743 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:39.013409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.012654 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub/model-serving-api-75bd89fdfc-mq8tg" podUID="2e100b28-7be2-4dcf-abbc-3532f79ff178" containerName="server" probeResult="failure" output="Get \"https://10.133.0.41:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:39.066407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.064926 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:16:39.156471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.154122 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:39.321824 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.321767 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:39.416434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.344567 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:39.417079 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.344611 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:39.428720 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.344647 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:39.429333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.376914 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:39.429778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.376962 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:16:39.430185 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.376984 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:16:39.435407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.383303 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:39.466098 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:16:39.423672 4457 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.505s" Nov 03 20:16:39.466098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.460799 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:16:39.466098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.460871 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:16:39.466098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.460924 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:16:39.466098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.460979 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:16:39.466098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.461029 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:16:39.466098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.461063 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:16:39.466098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.461110 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:16:39.466098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.461147 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:16:39.499560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.499527 4457 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 20:16:39.526645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.526612 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:16:39.556762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:39.556718 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:40.041424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.010582 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:16:40.143369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.140907 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:16:40.143369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.141023 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:16:40.143369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.141063 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:16:40.143369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.141126 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:16:40.143369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.141158 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:16:40.143369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.141206 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:16:40.148687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.148659 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:16:40.149353 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.149331 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:16:40.248416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.195084 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:40.248416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.195708 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:16:40.450406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.444565 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:16:40.741650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.741539 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:16:40.752412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.752340 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:16:40.794826 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.794794 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:16:40.848890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.848841 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz"] Nov 03 20:16:40.917665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:40.917631 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c847b846-98c4694d7-kk7gz"] Nov 03 20:16:41.795400 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:41.795329 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a708372-3610-4b08-b671-c76e6ef46675" path="/var/lib/kubelet/pods/8a708372-3610-4b08-b671-c76e6ef46675/volumes" Nov 03 20:16:43.120491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:43.120453 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf"] Nov 03 20:16:43.126902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:43.126877 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-e44e2278-5rdmf"] Nov 03 20:16:43.330967 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:43.329365 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h"] Nov 03 20:16:43.330967 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:43.329664 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" podUID="cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" containerName="istio-proxy" containerID="cri-o://9d6a14f62577806887dae01e44049d82d94da6a70ba099ecc7eb853dbb4c1818" gracePeriod=30 Nov 03 20:16:43.798151 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:43.797544 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31d8a65-79af-484c-8628-b5fcf2e26158" path="/var/lib/kubelet/pods/b31d8a65-79af-484c-8628-b5fcf2e26158/volumes" Nov 03 20:16:46.420659 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.420620 4457 generic.go:356] "Generic (PLEG): container finished" podID="cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" containerID="9d6a14f62577806887dae01e44049d82d94da6a70ba099ecc7eb853dbb4c1818" exitCode=0 Nov 03 20:16:46.421061 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.420771 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" event={"ID":"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd","Type":"ContainerDied","Data":"9d6a14f62577806887dae01e44049d82d94da6a70ba099ecc7eb853dbb4c1818"} Nov 03 20:16:46.444130 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.444098 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:16:46.586371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.586278 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-certs\") pod \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " Nov 03 20:16:46.586599 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.586411 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-token\" (UniqueName: \"kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-token\") pod \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " Nov 03 20:16:46.586599 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.586526 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-podinfo\") pod \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " Nov 03 20:16:46.586717 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.586639 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-socket\") pod \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " Nov 03 20:16:46.586803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.586714 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-credential-socket\") pod \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " Nov 03 20:16:46.586854 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.586837 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istiod-ca-cert\") pod \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " Nov 03 20:16:46.586904 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.586888 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-data\") pod \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " Nov 03 20:16:46.587067 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.587021 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-kube-api-access-xqmzj\" (UniqueName: \"kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-kube-api-access-xqmzj\") pod \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " Nov 03 20:16:46.587146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.587106 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-envoy\") pod \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\" (UID: \"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd\") " Nov 03 20:16:46.588426 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.588398 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-socket" pod "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" (UID: "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:46.588827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.588788 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-data" pod "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" (UID: "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:46.589277 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.589248 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-certs" pod "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" (UID: "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:46.589432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.589173 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-credential-socket" pod "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" (UID: "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:46.589686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.589659 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istiod-ca-cert" pod "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" (UID: "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:16:46.590885 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.590857 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-envoy" pod "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" (UID: "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:16:46.591563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.591366 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-kube-api-access-xqmzj" pod "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" (UID: "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd"). InnerVolumeSpecName "kube-api-access-xqmzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:46.591952 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.591919 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-token" pod "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" (UID: "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:16:46.592157 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.592124 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-podinfo" pod "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" (UID: "cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:16:46.689654 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.689615 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.689654 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.689649 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.689654 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.689660 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqmzj\" (UniqueName: \"kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-kube-api-access-xqmzj\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.689902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.689671 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.689902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.689680 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.689902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.689688 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.689902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.689696 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.689902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.689704 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:46.689902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:46.689712 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:16:47.432440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:47.432409 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" Nov 03 20:16:47.432943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:47.432425 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h" event={"ID":"cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd","Type":"ContainerDied","Data":"4c24f7cfe2516909e60910e1a71d5c06efb461922ceff1b07ce35c2d407cd2e6"} Nov 03 20:16:47.432943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:47.432533 4457 scope.go:122] "RemoveContainer" containerID="9d6a14f62577806887dae01e44049d82d94da6a70ba099ecc7eb853dbb4c1818" Nov 03 20:16:47.463718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:47.463681 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h"] Nov 03 20:16:47.470250 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:47.470215 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-e44e2278-gw-openshift-default-69d46c4b8f-kkg2h"] Nov 03 20:16:47.770112 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:16:47.770030 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" path="/var/lib/kubelet/pods/cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd/volumes" Nov 03 20:17:01.999410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:01.994043 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:17:02.035410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:02.034542 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:17:02.084793 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:02.084749 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:17:02.136171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:02.136139 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:17:06.165064 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:17:06.165015 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:17:08.155258 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.155222 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl"] Nov 03 20:17:08.156244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156213 4457 cpu_manager.go:414] "containerMap: removing container" podUID="8a708372-3610-4b08-b671-c76e6ef46675" containerName="maas-api" Nov 03 20:17:08.156244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156242 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a708372-3610-4b08-b671-c76e6ef46675" containerName="maas-api" Nov 03 20:17:08.156450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156298 4457 cpu_manager.go:414] "containerMap: removing container" podUID="cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" containerName="istio-proxy" Nov 03 20:17:08.156450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156306 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" containerName="istio-proxy" Nov 03 20:17:08.156450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156326 4457 cpu_manager.go:414] "containerMap: removing container" podUID="79caffd6-766b-4a72-951d-901ee5497282" containerName="payload-processing" Nov 03 20:17:08.156450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156335 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79caffd6-766b-4a72-951d-901ee5497282" containerName="payload-processing" Nov 03 20:17:08.156450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156349 4457 cpu_manager.go:414] "containerMap: removing container" podUID="9d3f423c-4451-4f01-8be1-44bf84c13e9a" containerName="payload-pre-processing" Nov 03 20:17:08.156450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156357 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d3f423c-4451-4f01-8be1-44bf84c13e9a" containerName="payload-pre-processing" Nov 03 20:17:08.156450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156374 4457 cpu_manager.go:414] "containerMap: removing container" podUID="b31d8a65-79af-484c-8628-b5fcf2e26158" containerName="revoke-keys" Nov 03 20:17:08.156450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156382 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31d8a65-79af-484c-8628-b5fcf2e26158" containerName="revoke-keys" Nov 03 20:17:08.156839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156587 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf9452b0-c6bf-4e5b-b46b-9985dc24a9fd" containerName="istio-proxy" Nov 03 20:17:08.156839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156601 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="79caffd6-766b-4a72-951d-901ee5497282" containerName="payload-processing" Nov 03 20:17:08.156839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156613 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d3f423c-4451-4f01-8be1-44bf84c13e9a" containerName="payload-pre-processing" Nov 03 20:17:08.156839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156624 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31d8a65-79af-484c-8628-b5fcf2e26158" containerName="revoke-keys" Nov 03 20:17:08.156839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.156638 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a708372-3610-4b08-b671-c76e6ef46675" containerName="maas-api" Nov 03 20:17:08.162700 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.162675 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" Nov 03 20:17:08.171410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.169900 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:17:08.171410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.169948 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:17:08.175255 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.175229 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl"] Nov 03 20:17:08.175756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.175725 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl\" (UID: \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" Nov 03 20:17:08.176068 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.176044 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl\" (UID: \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" Nov 03 20:17:08.176204 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.176187 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8czv\" (UniqueName: \"kubernetes.io/projected/bb75c330-e745-4e1f-83c6-2f45fdf147fb-kube-api-access-w8czv\") pod \"maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl\" (UID: \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" Nov 03 20:17:08.277694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.277646 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl\" (UID: \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" Nov 03 20:17:08.277920 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.277819 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl\" (UID: \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" Nov 03 20:17:08.277920 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.277854 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8czv\" (UniqueName: \"kubernetes.io/projected/bb75c330-e745-4e1f-83c6-2f45fdf147fb-kube-api-access-w8czv\") pod \"maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl\" (UID: \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" Nov 03 20:17:08.278596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.278553 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl\" (UID: \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" Nov 03 20:17:08.280468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.280441 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl\" (UID: \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" Nov 03 20:17:08.291319 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.291280 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8czv\" (UniqueName: \"kubernetes.io/projected/bb75c330-e745-4e1f-83c6-2f45fdf147fb-kube-api-access-w8czv\") pod \"maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl\" (UID: \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" Nov 03 20:17:08.484275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.484204 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" Nov 03 20:17:08.784380 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:17:08.784346 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb75c330_e745_4e1f_83c6_2f45fdf147fb.slice/crio-485e78c57b3f537ffc001106563345cdc7add2ab2c7c96f1d488ff3d5c069380 WatchSource:0}: Error finding container 485e78c57b3f537ffc001106563345cdc7add2ab2c7c96f1d488ff3d5c069380: Status 404 returned error can't find the container with id 485e78c57b3f537ffc001106563345cdc7add2ab2c7c96f1d488ff3d5c069380 Nov 03 20:17:08.785990 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:08.785964 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl"] Nov 03 20:17:09.064217 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:09.064184 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" event={"ID":"bb75c330-e745-4e1f-83c6-2f45fdf147fb","Type":"ContainerStarted","Data":"485e78c57b3f537ffc001106563345cdc7add2ab2c7c96f1d488ff3d5c069380"} Nov 03 20:17:10.079330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:10.079296 4457 generic.go:356] "Generic (PLEG): container finished" podID="bb75c330-e745-4e1f-83c6-2f45fdf147fb" containerID="54474491632b60ee9dad98d39fdb72cdb4661fd9e87e865f3060eff7f5c71abd" exitCode=0 Nov 03 20:17:10.079840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:10.079427 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" event={"ID":"bb75c330-e745-4e1f-83c6-2f45fdf147fb","Type":"ContainerDied","Data":"54474491632b60ee9dad98d39fdb72cdb4661fd9e87e865f3060eff7f5c71abd"} Nov 03 20:17:11.255130 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:11.255107 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" Nov 03 20:17:11.325991 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:11.325950 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb75c330-e745-4e1f-83c6-2f45fdf147fb-kube-api-access-w8czv\" (UniqueName: \"kubernetes.io/projected/bb75c330-e745-4e1f-83c6-2f45fdf147fb-kube-api-access-w8czv\") pod \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\" (UID: \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\") " Nov 03 20:17:11.326185 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:11.326075 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-openshift-ca\") pod \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\" (UID: \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\") " Nov 03 20:17:11.326296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:11.326273 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-xks-ca\") pod \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\" (UID: \"bb75c330-e745-4e1f-83c6-2f45fdf147fb\") " Nov 03 20:17:11.326809 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:11.326785 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-openshift-ca" pod "bb75c330-e745-4e1f-83c6-2f45fdf147fb" (UID: "bb75c330-e745-4e1f-83c6-2f45fdf147fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:11.327067 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:11.327048 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:11.328837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:11.328816 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-xks-ca" pod "bb75c330-e745-4e1f-83c6-2f45fdf147fb" (UID: "bb75c330-e745-4e1f-83c6-2f45fdf147fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:17:11.328937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:11.328847 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb75c330-e745-4e1f-83c6-2f45fdf147fb-kube-api-access-w8czv" pod "bb75c330-e745-4e1f-83c6-2f45fdf147fb" (UID: "bb75c330-e745-4e1f-83c6-2f45fdf147fb"). InnerVolumeSpecName "kube-api-access-w8czv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:11.428818 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:11.428727 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb75c330-e745-4e1f-83c6-2f45fdf147fb-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:11.428818 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:11.428761 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8czv\" (UniqueName: \"kubernetes.io/projected/bb75c330-e745-4e1f-83c6-2f45fdf147fb-kube-api-access-w8czv\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:12.102229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:12.102137 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" Nov 03 20:17:12.102421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:12.102127 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl" event={"ID":"bb75c330-e745-4e1f-83c6-2f45fdf147fb","Type":"ContainerDied","Data":"485e78c57b3f537ffc001106563345cdc7add2ab2c7c96f1d488ff3d5c069380"} Nov 03 20:17:12.102421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:12.102278 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="485e78c57b3f537ffc001106563345cdc7add2ab2c7c96f1d488ff3d5c069380" Nov 03 20:17:13.043537 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:13.043501 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5"] Nov 03 20:17:13.048698 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:13.048674 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-727f0656-kk7c5"] Nov 03 20:17:13.765434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:13.761727 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b11d16e-96b3-43c4-8a74-2d3536a42f67" path="/var/lib/kubelet/pods/2b11d16e-96b3-43c4-8a74-2d3536a42f67/volumes" Nov 03 20:17:14.043536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:14.043446 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c"] Nov 03 20:17:14.048572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:14.048551 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-247ddf7f-r9j7c"] Nov 03 20:17:15.762471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:15.762368 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="669b4131-2371-4b01-b932-7035e230d493" path="/var/lib/kubelet/pods/669b4131-2371-4b01-b932-7035e230d493/volumes" Nov 03 20:17:20.416464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.416428 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9"] Nov 03 20:17:20.417094 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.417079 4457 cpu_manager.go:414] "containerMap: removing container" podUID="bb75c330-e745-4e1f-83c6-2f45fdf147fb" containerName="revoke-keys" Nov 03 20:17:20.417156 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.417094 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb75c330-e745-4e1f-83c6-2f45fdf147fb" containerName="revoke-keys" Nov 03 20:17:20.417225 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.417216 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb75c330-e745-4e1f-83c6-2f45fdf147fb" containerName="revoke-keys" Nov 03 20:17:20.426972 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.426942 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" Nov 03 20:17:20.429310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.429280 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9"] Nov 03 20:17:20.431578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.431550 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:17:20.432295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.432272 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:17:20.515854 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.515587 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvddd\" (UniqueName: \"kubernetes.io/projected/792d1bd0-e350-4439-be6b-e49065d53e68-kube-api-access-dvddd\") pod \"maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9\" (UID: \"792d1bd0-e350-4439-be6b-e49065d53e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" Nov 03 20:17:20.515854 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.515834 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9\" (UID: \"792d1bd0-e350-4439-be6b-e49065d53e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" Nov 03 20:17:20.516152 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.515887 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9\" (UID: \"792d1bd0-e350-4439-be6b-e49065d53e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" Nov 03 20:17:20.617441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.617407 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9\" (UID: \"792d1bd0-e350-4439-be6b-e49065d53e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" Nov 03 20:17:20.617620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.617462 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9\" (UID: \"792d1bd0-e350-4439-be6b-e49065d53e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" Nov 03 20:17:20.617620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.617538 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvddd\" (UniqueName: \"kubernetes.io/projected/792d1bd0-e350-4439-be6b-e49065d53e68-kube-api-access-dvddd\") pod \"maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9\" (UID: \"792d1bd0-e350-4439-be6b-e49065d53e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" Nov 03 20:17:20.618180 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.618153 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9\" (UID: \"792d1bd0-e350-4439-be6b-e49065d53e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" Nov 03 20:17:20.620019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.619996 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9\" (UID: \"792d1bd0-e350-4439-be6b-e49065d53e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" Nov 03 20:17:20.630519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.630493 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvddd\" (UniqueName: \"kubernetes.io/projected/792d1bd0-e350-4439-be6b-e49065d53e68-kube-api-access-dvddd\") pod \"maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9\" (UID: \"792d1bd0-e350-4439-be6b-e49065d53e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" Nov 03 20:17:20.804716 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:20.804682 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" Nov 03 20:17:21.097009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:21.096974 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9"] Nov 03 20:17:21.100333 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:17:21.100298 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod792d1bd0_e350_4439_be6b_e49065d53e68.slice/crio-e79777c1d04ebd132ffaea1674720ff8c89503c0421409fff7979089d52a56e6 WatchSource:0}: Error finding container e79777c1d04ebd132ffaea1674720ff8c89503c0421409fff7979089d52a56e6: Status 404 returned error can't find the container with id e79777c1d04ebd132ffaea1674720ff8c89503c0421409fff7979089d52a56e6 Nov 03 20:17:21.271412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:21.270423 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" event={"ID":"792d1bd0-e350-4439-be6b-e49065d53e68","Type":"ContainerStarted","Data":"e79777c1d04ebd132ffaea1674720ff8c89503c0421409fff7979089d52a56e6"} Nov 03 20:17:21.827784 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:21.827551 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl"] Nov 03 20:17:21.866252 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:21.866216 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-11b0d340-zqgsl"] Nov 03 20:17:22.047565 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:22.047528 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r"] Nov 03 20:17:22.057427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:22.057300 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06c5e3fd-5kp9r"] Nov 03 20:17:22.319636 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:22.319600 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" event={"ID":"792d1bd0-e350-4439-be6b-e49065d53e68","Type":"ContainerStarted","Data":"e697cf117650a9f3fb8f42f1687eba15e51ddd784f6192bf580495359de7b37b"} Nov 03 20:17:23.492436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:23.491835 4457 generic.go:356] "Generic (PLEG): container finished" podID="792d1bd0-e350-4439-be6b-e49065d53e68" containerID="e697cf117650a9f3fb8f42f1687eba15e51ddd784f6192bf580495359de7b37b" exitCode=0 Nov 03 20:17:23.492436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:23.491992 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" event={"ID":"792d1bd0-e350-4439-be6b-e49065d53e68","Type":"ContainerDied","Data":"e697cf117650a9f3fb8f42f1687eba15e51ddd784f6192bf580495359de7b37b"} Nov 03 20:17:23.790771 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:23.789181 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="993aa66d-9758-47e9-b83c-9341d0e26995" path="/var/lib/kubelet/pods/993aa66d-9758-47e9-b83c-9341d0e26995/volumes" Nov 03 20:17:23.790771 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:23.790297 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb75c330-e745-4e1f-83c6-2f45fdf147fb" path="/var/lib/kubelet/pods/bb75c330-e745-4e1f-83c6-2f45fdf147fb/volumes" Nov 03 20:17:23.836640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:23.836607 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" Nov 03 20:17:23.952875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:23.949087 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-xks-ca\") pod \"792d1bd0-e350-4439-be6b-e49065d53e68\" (UID: \"792d1bd0-e350-4439-be6b-e49065d53e68\") " Nov 03 20:17:23.952875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:23.949245 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/792d1bd0-e350-4439-be6b-e49065d53e68-kube-api-access-dvddd\" (UniqueName: \"kubernetes.io/projected/792d1bd0-e350-4439-be6b-e49065d53e68-kube-api-access-dvddd\") pod \"792d1bd0-e350-4439-be6b-e49065d53e68\" (UID: \"792d1bd0-e350-4439-be6b-e49065d53e68\") " Nov 03 20:17:23.952875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:23.949549 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-openshift-ca\") pod \"792d1bd0-e350-4439-be6b-e49065d53e68\" (UID: \"792d1bd0-e350-4439-be6b-e49065d53e68\") " Nov 03 20:17:23.953737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:23.953364 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-openshift-ca" pod "792d1bd0-e350-4439-be6b-e49065d53e68" (UID: "792d1bd0-e350-4439-be6b-e49065d53e68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:17:23.959278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:23.959249 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/792d1bd0-e350-4439-be6b-e49065d53e68-kube-api-access-dvddd" pod "792d1bd0-e350-4439-be6b-e49065d53e68" (UID: "792d1bd0-e350-4439-be6b-e49065d53e68"). InnerVolumeSpecName "kube-api-access-dvddd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:17:23.962508 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:23.962480 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-xks-ca" pod "792d1bd0-e350-4439-be6b-e49065d53e68" (UID: "792d1bd0-e350-4439-be6b-e49065d53e68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:17:24.051755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:24.051725 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:24.051755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:24.051755 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/792d1bd0-e350-4439-be6b-e49065d53e68-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:24.052021 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:24.051768 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvddd\" (UniqueName: \"kubernetes.io/projected/792d1bd0-e350-4439-be6b-e49065d53e68-kube-api-access-dvddd\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:17:24.506491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:24.505888 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" Nov 03 20:17:24.506491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:24.505908 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9" event={"ID":"792d1bd0-e350-4439-be6b-e49065d53e68","Type":"ContainerDied","Data":"e79777c1d04ebd132ffaea1674720ff8c89503c0421409fff7979089d52a56e6"} Nov 03 20:17:24.506491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:24.505975 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e79777c1d04ebd132ffaea1674720ff8c89503c0421409fff7979089d52a56e6" Nov 03 20:17:25.032021 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:25.031988 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd"] Nov 03 20:17:25.041305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:25.041275 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60d33091-7thdd"] Nov 03 20:17:25.778220 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:25.778171 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8625493a-9238-4f7a-86cb-320fd834d9e0" path="/var/lib/kubelet/pods/8625493a-9238-4f7a-86cb-320fd834d9e0/volumes" Nov 03 20:17:28.048760 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:28.047188 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52"] Nov 03 20:17:28.055048 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:28.055017 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e563a98-h9k52"] Nov 03 20:17:29.042740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:29.042694 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf"] Nov 03 20:17:29.047737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:29.047710 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-285d4478-zd4rf"] Nov 03 20:17:29.777709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:29.777672 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536d878c-d7a4-43ba-bcbb-65ab24fb30c4" path="/var/lib/kubelet/pods/536d878c-d7a4-43ba-bcbb-65ab24fb30c4/volumes" Nov 03 20:17:29.779170 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:29.779139 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1" path="/var/lib/kubelet/pods/c2fe4f47-0d4f-4fd8-9a7f-c61225c80cb1/volumes" Nov 03 20:17:35.989600 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:35.989501 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9"] Nov 03 20:17:35.997593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:35.997561 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ffd1cf5a-mwvz9"] Nov 03 20:17:36.382193 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:17:36.382156 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:17:37.030817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:37.030777 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr"] Nov 03 20:17:37.037294 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:37.037259 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-949440a8-wtmdr"] Nov 03 20:17:37.872516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:37.869725 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="792d1bd0-e350-4439-be6b-e49065d53e68" path="/var/lib/kubelet/pods/792d1bd0-e350-4439-be6b-e49065d53e68/volumes" Nov 03 20:17:37.879561 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:37.879487 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98df7398-6420-4228-b013-94e01b98bccc" path="/var/lib/kubelet/pods/98df7398-6420-4228-b013-94e01b98bccc/volumes" Nov 03 20:17:40.029458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:40.029424 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5"] Nov 03 20:17:40.034771 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:40.034739 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7f02c74-kbfq5"] Nov 03 20:17:41.772515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:41.772478 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9196fff0-b36c-4ff0-a620-fb1e586b6caf" path="/var/lib/kubelet/pods/9196fff0-b36c-4ff0-a620-fb1e586b6caf/volumes" Nov 03 20:17:50.012228 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.011926 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm"] Nov 03 20:17:50.013211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.013183 4457 cpu_manager.go:414] "containerMap: removing container" podUID="792d1bd0-e350-4439-be6b-e49065d53e68" containerName="revoke-keys" Nov 03 20:17:50.013558 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.013513 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="792d1bd0-e350-4439-be6b-e49065d53e68" containerName="revoke-keys" Nov 03 20:17:50.013938 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.013916 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="792d1bd0-e350-4439-be6b-e49065d53e68" containerName="revoke-keys" Nov 03 20:17:50.042529 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.042498 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm"] Nov 03 20:17:50.042738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.042618 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.050278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.050251 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-b35c30-serving-cert\"" Nov 03 20:17:50.170119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.170079 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dfce2c-4a66-479e-afcf-ddce81a9252b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.170288 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.170245 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kzf2\" (UniqueName: \"kubernetes.io/projected/b1dfce2c-4a66-479e-afcf-ddce81a9252b-kube-api-access-5kzf2\") pod \"maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.170433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.170350 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.170433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.170428 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.272525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.272431 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kzf2\" (UniqueName: \"kubernetes.io/projected/b1dfce2c-4a66-479e-afcf-ddce81a9252b-kube-api-access-5kzf2\") pod \"maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.272719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.272563 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.272719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.272618 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.272834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.272763 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dfce2c-4a66-479e-afcf-ddce81a9252b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.273498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.273472 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dfce2c-4a66-479e-afcf-ddce81a9252b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.276550 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.276526 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.276661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.276645 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.291761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.291730 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kzf2\" (UniqueName: \"kubernetes.io/projected/b1dfce2c-4a66-479e-afcf-ddce81a9252b-kube-api-access-5kzf2\") pod \"maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.372025 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.371984 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:50.739948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.739912 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm"] Nov 03 20:17:50.986448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:50.979637 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" event={"ID":"b1dfce2c-4a66-479e-afcf-ddce81a9252b","Type":"ContainerStarted","Data":"f9ec585a087f49a1a7dfd812a0dfb0826e03b5c93d9750436f648b195771b497"} Nov 03 20:17:51.991640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:51.991602 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" event={"ID":"b1dfce2c-4a66-479e-afcf-ddce81a9252b","Type":"ContainerStarted","Data":"9d32303bd0b4291937f0df403840ef93fd12f5181839b0038ae162cb619e156b"} Nov 03 20:17:51.992108 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:51.991691 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:52.014902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:52.014847 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" podStartSLOduration=2.462531584 podStartE2EDuration="3.014832703s" totalImagesPullingTime="552.301119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:52.011668505 +0000 UTC m=+1250.387265253" watchObservedRunningTime="2026-11-03 20:17:52.014832703 +0000 UTC m=+1250.390429451" Nov 03 20:17:53.010409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.010063 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857"] Nov 03 20:17:53.016714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.016687 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.037769 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.037241 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857"] Nov 03 20:17:53.066866 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.066825 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-1c6a98-openshift-default-dockercfg-6qxq9\"" Nov 03 20:17:53.071894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.071867 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr"] Nov 03 20:17:53.077315 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.077277 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2495bf59-m99fr"] Nov 03 20:17:53.078220 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.078197 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-podinfo\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.078468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.078287 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-data\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.078468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.078353 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-credential-socket\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.078468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.078425 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24jqv\" (UniqueName: \"kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-kube-api-access-24jqv\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.078649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.078474 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-socket\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.078649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.078536 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-envoy\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.078756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.078648 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-certs\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.078756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.078745 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-token\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.078857 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.078800 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b8b440-7a7c-4713-bce7-529cf543c29d-istiod-ca-cert\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.194458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.194408 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24jqv\" (UniqueName: \"kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-kube-api-access-24jqv\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.225414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.207519 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-socket\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.225414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.207628 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-envoy\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.225414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.207797 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-certs\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.225414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.207914 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-token\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.225414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.207960 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b8b440-7a7c-4713-bce7-529cf543c29d-istiod-ca-cert\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.225414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.208048 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-podinfo\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.225414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.208182 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-data\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.225414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.208287 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-credential-socket\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.225414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.209198 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-socket\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.225414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.220185 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-certs\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.225414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.221719 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-data\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.227084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.227054 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b8b440-7a7c-4713-bce7-529cf543c29d-istiod-ca-cert\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.251868 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.237808 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-credential-socket\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.252089 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.250687 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-envoy\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.266457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.252847 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-podinfo\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.286431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.282678 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24jqv\" (UniqueName: \"kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-kube-api-access-24jqv\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.310550 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.308760 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-token\") pod \"e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.361558 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.361183 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:53.743981 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.743945 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857"] Nov 03 20:17:53.769825 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.761355 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:53.769825 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.761457 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:53.769825 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.761501 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:17:53.772676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:53.772332 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a305e5f2-c91f-44fd-9568-150cbdda93da" path="/var/lib/kubelet/pods/a305e5f2-c91f-44fd-9568-150cbdda93da/volumes" Nov 03 20:17:54.046460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:54.045893 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" event={"ID":"a6b8b440-7a7c-4713-bce7-529cf543c29d","Type":"ContainerStarted","Data":"39f18f1f4d7bd1e8eed7adb3f8bb1f04487d8c14fdeb36336acd37e30985b78e"} Nov 03 20:17:54.046460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:54.045949 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" event={"ID":"a6b8b440-7a7c-4713-bce7-529cf543c29d","Type":"ContainerStarted","Data":"56d34024a0cc1cceb6b24a551f8ee027b010fe3ef0232765592c3c58c1d94a65"} Nov 03 20:17:54.075087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:54.074949 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" podStartSLOduration=2.074930215 podStartE2EDuration="2.074930215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:54.06873162 +0000 UTC m=+1252.444328380" watchObservedRunningTime="2026-11-03 20:17:54.074930215 +0000 UTC m=+1252.450526963" Nov 03 20:17:54.365471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:54.361361 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:54.368779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:54.368751 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:55.057061 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:55.057008 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:55.058278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:55.058255 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:17:57.917666 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:57.917631 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q"] Nov 03 20:17:57.923212 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:57.923190 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" Nov 03 20:17:57.935484 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:57.935459 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q"] Nov 03 20:17:57.967685 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:57.967647 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-1c6a98\"" Nov 03 20:17:57.996750 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:57.996717 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-1c6a98-dockercfg-z8mjm\"" Nov 03 20:17:58.018498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:58.018458 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q\" (UID: \"df7f2048-8b4e-46ef-a5b9-e2504a0be18c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" Nov 03 20:17:58.022410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:58.019065 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xjwg\" (UniqueName: \"kubernetes.io/projected/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-kube-api-access-4xjwg\") pod \"payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q\" (UID: \"df7f2048-8b4e-46ef-a5b9-e2504a0be18c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" Nov 03 20:17:58.156768 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:58.156719 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xjwg\" (UniqueName: \"kubernetes.io/projected/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-kube-api-access-4xjwg\") pod \"payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q\" (UID: \"df7f2048-8b4e-46ef-a5b9-e2504a0be18c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" Nov 03 20:17:58.157056 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:58.156915 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q\" (UID: \"df7f2048-8b4e-46ef-a5b9-e2504a0be18c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" Nov 03 20:17:58.157792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:58.157768 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:17:58.235562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:58.235481 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q\" (UID: \"df7f2048-8b4e-46ef-a5b9-e2504a0be18c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" Nov 03 20:17:58.273166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:58.273136 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xjwg\" (UniqueName: \"kubernetes.io/projected/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-kube-api-access-4xjwg\") pod \"payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q\" (UID: \"df7f2048-8b4e-46ef-a5b9-e2504a0be18c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" Nov 03 20:17:58.554954 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:58.554925 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" Nov 03 20:17:58.876804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:58.876773 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q"] Nov 03 20:17:58.877847 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:17:58.877818 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7f2048_8b4e_46ef_a5b9_e2504a0be18c.slice/crio-b6e93d570f504aae13a190dbb8269b0b2fcfbf9724630fa9852e9162eae4f8ca WatchSource:0}: Error finding container b6e93d570f504aae13a190dbb8269b0b2fcfbf9724630fa9852e9162eae4f8ca: Status 404 returned error can't find the container with id b6e93d570f504aae13a190dbb8269b0b2fcfbf9724630fa9852e9162eae4f8ca Nov 03 20:17:59.311489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:59.311452 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" event={"ID":"df7f2048-8b4e-46ef-a5b9-e2504a0be18c","Type":"ContainerStarted","Data":"3e262bb946869e428be51ab90c73d5c5ab96ddb937d4e42881e5824bb0c5f5e9"} Nov 03 20:17:59.311960 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:59.311512 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" event={"ID":"df7f2048-8b4e-46ef-a5b9-e2504a0be18c","Type":"ContainerStarted","Data":"b6e93d570f504aae13a190dbb8269b0b2fcfbf9724630fa9852e9162eae4f8ca"} Nov 03 20:17:59.311960 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:59.311568 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" Nov 03 20:17:59.335490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:17:59.335438 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" podStartSLOduration=2.335422546 podStartE2EDuration="2.335422546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:17:59.329659199 +0000 UTC m=+1257.705255956" watchObservedRunningTime="2026-11-03 20:17:59.335422546 +0000 UTC m=+1257.711019291" Nov 03 20:18:06.152296 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:18:06.152248 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:18:10.334743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:10.334713 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" Nov 03 20:18:36.193146 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:18:36.193076 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:18:41.446449 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:41.446379 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:18:41.542529 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:41.542475 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:41.737471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:41.735510 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:41.774964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:41.773682 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:18:41.774964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:41.774132 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:18:41.775628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:41.775478 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:41.775628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:41.775579 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:41.854031 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:41.851630 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:41.929413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:41.928571 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.019472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.018710 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.127456 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.119359 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.149418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.145513 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.174412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.168740 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.450411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.449869 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.450937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.450456 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:18:42.772515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.770539 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.772515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.771078 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded" Nov 03 20:18:42.772515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.772418 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:18:42.775693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.772909 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:18:42.775693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.773873 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:42.775693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.775001 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:18:42.856959 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.856909 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:18:42.858171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.858145 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:18:42.951407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.931232 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:18:42.951407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:42.934052 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:18:43.025190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:43.025072 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:18:43.037231 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:43.036471 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:18:43.038161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:43.037577 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:18:43.040590 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:43.040567 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:18:43.055369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:43.055339 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:18:43.117740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:43.103903 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:18:43.117740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:43.105788 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:18:43.117740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:43.105956 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:18:43.201609 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:43.200271 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:18:44.519826 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.519776 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l"] Nov 03 20:18:44.538060 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.538024 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l"] Nov 03 20:18:44.538401 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.538327 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.545533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.545504 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7ea43c-openshift-default-dockercfg-k4lqs\"" Nov 03 20:18:44.712472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.686599 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-data\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.712902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.712868 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-podinfo\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.775462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.718516 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-envoy\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.775462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.718643 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-credential-socket\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.775462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.718817 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-certs\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.775462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.718874 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2qvf\" (UniqueName: \"kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-kube-api-access-z2qvf\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.775462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.718973 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90af0b2d-03b0-4532-85ee-f3282cc9637a-istiod-ca-cert\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.775462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.719058 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-socket\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.775462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.719160 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-token\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.862410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.835618 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-token\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.874548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.874508 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-data\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.902755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.875924 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-data\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.903019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.902987 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-podinfo\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.930263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.903931 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-envoy\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.930263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.920915 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-credential-socket\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.930263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.921369 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-credential-socket\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.930263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.920827 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-token\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.930263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.922317 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-certs\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.930263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.922880 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2qvf\" (UniqueName: \"kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-kube-api-access-z2qvf\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.930263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.930196 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-envoy\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.932000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.931766 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90af0b2d-03b0-4532-85ee-f3282cc9637a-istiod-ca-cert\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.937409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.932455 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-certs\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.937409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.932541 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-socket\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.937409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.932829 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-socket\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.937409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.933775 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90af0b2d-03b0-4532-85ee-f3282cc9637a-istiod-ca-cert\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.937409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.935222 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-podinfo\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:44.954436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:44.953908 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2qvf\" (UniqueName: \"kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-kube-api-access-z2qvf\") pod \"e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:45.164405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:45.163057 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:45.550043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:45.549720 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l"] Nov 03 20:18:45.553276 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:18:45.553242 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90af0b2d_03b0_4532_85ee_f3282cc9637a.slice/crio-2710ed5ab459be5ae49b231e2a3e0ea3fe11616c3a04c9e90c64dec6de168cbd WatchSource:0}: Error finding container 2710ed5ab459be5ae49b231e2a3e0ea3fe11616c3a04c9e90c64dec6de168cbd: Status 404 returned error can't find the container with id 2710ed5ab459be5ae49b231e2a3e0ea3fe11616c3a04c9e90c64dec6de168cbd Nov 03 20:18:45.556545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:45.556498 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:45.556658 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:45.556600 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:45.556658 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:45.556647 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:46.168772 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.168732 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d"] Nov 03 20:18:46.174471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.174268 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" Nov 03 20:18:46.181034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.180962 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:18:46.181034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.181006 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:18:46.184410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.184368 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d"] Nov 03 20:18:46.263567 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.263533 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ddaa2d-106d-4537-8421-130805474905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ccdee44-vgj4d\" (UID: \"96ddaa2d-106d-4537-8421-130805474905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" Nov 03 20:18:46.263787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.263608 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl6n9\" (UniqueName: \"kubernetes.io/projected/96ddaa2d-106d-4537-8421-130805474905-kube-api-access-tl6n9\") pod \"maas-api-revoke-sub-9ccdee44-vgj4d\" (UID: \"96ddaa2d-106d-4537-8421-130805474905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" Nov 03 20:18:46.263787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.263772 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ddaa2d-106d-4537-8421-130805474905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ccdee44-vgj4d\" (UID: \"96ddaa2d-106d-4537-8421-130805474905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" Nov 03 20:18:46.364932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.364896 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ddaa2d-106d-4537-8421-130805474905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ccdee44-vgj4d\" (UID: \"96ddaa2d-106d-4537-8421-130805474905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" Nov 03 20:18:46.365130 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.364985 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl6n9\" (UniqueName: \"kubernetes.io/projected/96ddaa2d-106d-4537-8421-130805474905-kube-api-access-tl6n9\") pod \"maas-api-revoke-sub-9ccdee44-vgj4d\" (UID: \"96ddaa2d-106d-4537-8421-130805474905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" Nov 03 20:18:46.365130 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.365047 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ddaa2d-106d-4537-8421-130805474905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ccdee44-vgj4d\" (UID: \"96ddaa2d-106d-4537-8421-130805474905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" Nov 03 20:18:46.366111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.366083 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ddaa2d-106d-4537-8421-130805474905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ccdee44-vgj4d\" (UID: \"96ddaa2d-106d-4537-8421-130805474905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" Nov 03 20:18:46.384805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.384772 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ddaa2d-106d-4537-8421-130805474905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ccdee44-vgj4d\" (UID: \"96ddaa2d-106d-4537-8421-130805474905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" Nov 03 20:18:46.389102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.387122 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" event={"ID":"90af0b2d-03b0-4532-85ee-f3282cc9637a","Type":"ContainerStarted","Data":"9997f4ec20b8ebad6e4d59ad71e3d378339d5ca88f1b29495739890a9a50a094"} Nov 03 20:18:46.389102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.387187 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" event={"ID":"90af0b2d-03b0-4532-85ee-f3282cc9637a","Type":"ContainerStarted","Data":"2710ed5ab459be5ae49b231e2a3e0ea3fe11616c3a04c9e90c64dec6de168cbd"} Nov 03 20:18:46.389102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.387906 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl6n9\" (UniqueName: \"kubernetes.io/projected/96ddaa2d-106d-4537-8421-130805474905-kube-api-access-tl6n9\") pod \"maas-api-revoke-sub-9ccdee44-vgj4d\" (UID: \"96ddaa2d-106d-4537-8421-130805474905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" Nov 03 20:18:46.425603 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.425501 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" podStartSLOduration=2.425483305 podStartE2EDuration="2.425483305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:46.420887092 +0000 UTC m=+1304.796483841" watchObservedRunningTime="2026-11-03 20:18:46.425483305 +0000 UTC m=+1304.801080052" Nov 03 20:18:46.495351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.495308 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" Nov 03 20:18:46.913138 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:18:46.913102 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ddaa2d_106d_4537_8421_130805474905.slice/crio-0f742fb00ae7dfc82d784cd2419bced3d392d0f0163da8002357c158b79a3285 WatchSource:0}: Error finding container 0f742fb00ae7dfc82d784cd2419bced3d392d0f0163da8002357c158b79a3285: Status 404 returned error can't find the container with id 0f742fb00ae7dfc82d784cd2419bced3d392d0f0163da8002357c158b79a3285 Nov 03 20:18:46.913875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:46.913851 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d"] Nov 03 20:18:47.165133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:47.164186 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:47.176682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:47.175852 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:47.400293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:47.400247 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" event={"ID":"96ddaa2d-106d-4537-8421-130805474905","Type":"ContainerStarted","Data":"f48fef060688540d37cbbf95e84cb80d0e2eccb124aa109d6851b47726d103c6"} Nov 03 20:18:47.400565 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:47.400319 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" event={"ID":"96ddaa2d-106d-4537-8421-130805474905","Type":"ContainerStarted","Data":"0f742fb00ae7dfc82d784cd2419bced3d392d0f0163da8002357c158b79a3285"} Nov 03 20:18:47.401338 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:47.401312 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:47.402561 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:47.402540 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:18:47.425241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:47.425171 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" podStartSLOduration=1.425153265 podStartE2EDuration="1.425153265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:47.419418804 +0000 UTC m=+1305.795015550" watchObservedRunningTime="2026-11-03 20:18:47.425153265 +0000 UTC m=+1305.800750013" Nov 03 20:18:48.415399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:48.415291 4457 generic.go:356] "Generic (PLEG): container finished" podID="96ddaa2d-106d-4537-8421-130805474905" containerID="f48fef060688540d37cbbf95e84cb80d0e2eccb124aa109d6851b47726d103c6" exitCode=0 Nov 03 20:18:48.415399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:48.415370 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" event={"ID":"96ddaa2d-106d-4537-8421-130805474905","Type":"ContainerDied","Data":"f48fef060688540d37cbbf95e84cb80d0e2eccb124aa109d6851b47726d103c6"} Nov 03 20:18:49.756721 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:49.756690 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" Nov 03 20:18:49.854791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:49.854762 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96ddaa2d-106d-4537-8421-130805474905-kube-api-access-tl6n9\" (UniqueName: \"kubernetes.io/projected/96ddaa2d-106d-4537-8421-130805474905-kube-api-access-tl6n9\") pod \"96ddaa2d-106d-4537-8421-130805474905\" (UID: \"96ddaa2d-106d-4537-8421-130805474905\") " Nov 03 20:18:49.857038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:49.854846 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96ddaa2d-106d-4537-8421-130805474905-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ddaa2d-106d-4537-8421-130805474905-maas-api-xks-ca\") pod \"96ddaa2d-106d-4537-8421-130805474905\" (UID: \"96ddaa2d-106d-4537-8421-130805474905\") " Nov 03 20:18:49.857038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:49.854885 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96ddaa2d-106d-4537-8421-130805474905-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ddaa2d-106d-4537-8421-130805474905-maas-api-openshift-ca\") pod \"96ddaa2d-106d-4537-8421-130805474905\" (UID: \"96ddaa2d-106d-4537-8421-130805474905\") " Nov 03 20:18:49.857038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:49.855241 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ddaa2d-106d-4537-8421-130805474905-maas-api-openshift-ca" pod "96ddaa2d-106d-4537-8421-130805474905" (UID: "96ddaa2d-106d-4537-8421-130805474905"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:18:49.857038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:49.855739 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96ddaa2d-106d-4537-8421-130805474905-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:49.857580 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:49.857554 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ddaa2d-106d-4537-8421-130805474905-kube-api-access-tl6n9" pod "96ddaa2d-106d-4537-8421-130805474905" (UID: "96ddaa2d-106d-4537-8421-130805474905"). InnerVolumeSpecName "kube-api-access-tl6n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:18:49.857738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:49.857569 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96ddaa2d-106d-4537-8421-130805474905-maas-api-xks-ca" pod "96ddaa2d-106d-4537-8421-130805474905" (UID: "96ddaa2d-106d-4537-8421-130805474905"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:18:49.965838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:49.960244 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl6n9\" (UniqueName: \"kubernetes.io/projected/96ddaa2d-106d-4537-8421-130805474905-kube-api-access-tl6n9\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:49.965838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:49.960276 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96ddaa2d-106d-4537-8421-130805474905-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:18:50.497937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:50.497901 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" Nov 03 20:18:50.498244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:50.497911 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d" event={"ID":"96ddaa2d-106d-4537-8421-130805474905","Type":"ContainerDied","Data":"0f742fb00ae7dfc82d784cd2419bced3d392d0f0163da8002357c158b79a3285"} Nov 03 20:18:50.498367 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:50.498292 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f742fb00ae7dfc82d784cd2419bced3d392d0f0163da8002357c158b79a3285" Nov 03 20:18:50.510214 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:50.510183 4457 generic.go:356] "Generic (PLEG): container finished" podID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerID="39f18f1f4d7bd1e8eed7adb3f8bb1f04487d8c14fdeb36336acd37e30985b78e" exitCode=0 Nov 03 20:18:50.510496 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:50.510402 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" event={"ID":"a6b8b440-7a7c-4713-bce7-529cf543c29d","Type":"ContainerDied","Data":"39f18f1f4d7bd1e8eed7adb3f8bb1f04487d8c14fdeb36336acd37e30985b78e"} Nov 03 20:18:50.511001 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:50.510979 4457 scope.go:122] "RemoveContainer" containerID="39f18f1f4d7bd1e8eed7adb3f8bb1f04487d8c14fdeb36336acd37e30985b78e" Nov 03 20:18:50.512969 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:50.512938 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:50.513087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:50.513019 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:50.513087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:50.513055 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:18:51.240975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.240943 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t"] Nov 03 20:18:51.241675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.241659 4457 cpu_manager.go:414] "containerMap: removing container" podUID="96ddaa2d-106d-4537-8421-130805474905" containerName="revoke-keys" Nov 03 20:18:51.241675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.241676 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96ddaa2d-106d-4537-8421-130805474905" containerName="revoke-keys" Nov 03 20:18:51.241812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.241803 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="96ddaa2d-106d-4537-8421-130805474905" containerName="revoke-keys" Nov 03 20:18:51.245722 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.245701 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" Nov 03 20:18:51.251953 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.251928 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-15e02e6b\"" Nov 03 20:18:51.252076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.252022 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-15e02e6b-dockercfg-cqhtd\"" Nov 03 20:18:51.258609 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.258585 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t"] Nov 03 20:18:51.286819 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.286787 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn"] Nov 03 20:18:51.292888 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.292855 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" Nov 03 20:18:51.306261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.306231 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn"] Nov 03 20:18:51.313835 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.313461 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jk86\" (UniqueName: \"kubernetes.io/projected/c254c14f-13f1-4e20-bd27-14585726663f-kube-api-access-9jk86\") pod \"payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t\" (UID: \"c254c14f-13f1-4e20-bd27-14585726663f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" Nov 03 20:18:51.313835 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.313549 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c254c14f-13f1-4e20-bd27-14585726663f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t\" (UID: \"c254c14f-13f1-4e20-bd27-14585726663f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" Nov 03 20:18:51.430412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.415482 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jk86\" (UniqueName: \"kubernetes.io/projected/c254c14f-13f1-4e20-bd27-14585726663f-kube-api-access-9jk86\") pod \"payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t\" (UID: \"c254c14f-13f1-4e20-bd27-14585726663f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" Nov 03 20:18:51.430412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.415578 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c254c14f-13f1-4e20-bd27-14585726663f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t\" (UID: \"c254c14f-13f1-4e20-bd27-14585726663f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" Nov 03 20:18:51.430412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.415661 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b5ac0a-b306-458b-8b39-96deb3b575fa-plugins-config-volume\") pod \"payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn\" (UID: \"81b5ac0a-b306-458b-8b39-96deb3b575fa\") " pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" Nov 03 20:18:51.430412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.415740 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6rpn\" (UniqueName: \"kubernetes.io/projected/81b5ac0a-b306-458b-8b39-96deb3b575fa-kube-api-access-k6rpn\") pod \"payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn\" (UID: \"81b5ac0a-b306-458b-8b39-96deb3b575fa\") " pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" Nov 03 20:18:51.499467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.487812 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c254c14f-13f1-4e20-bd27-14585726663f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t\" (UID: \"c254c14f-13f1-4e20-bd27-14585726663f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" Nov 03 20:18:51.524422 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.501570 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jk86\" (UniqueName: \"kubernetes.io/projected/c254c14f-13f1-4e20-bd27-14585726663f-kube-api-access-9jk86\") pod \"payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t\" (UID: \"c254c14f-13f1-4e20-bd27-14585726663f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" Nov 03 20:18:51.561182 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.560976 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b5ac0a-b306-458b-8b39-96deb3b575fa-plugins-config-volume\") pod \"payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn\" (UID: \"81b5ac0a-b306-458b-8b39-96deb3b575fa\") " pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" Nov 03 20:18:51.561182 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.561106 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6rpn\" (UniqueName: \"kubernetes.io/projected/81b5ac0a-b306-458b-8b39-96deb3b575fa-kube-api-access-k6rpn\") pod \"payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn\" (UID: \"81b5ac0a-b306-458b-8b39-96deb3b575fa\") " pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" Nov 03 20:18:51.568512 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.564651 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" Nov 03 20:18:51.568512 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.565096 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b5ac0a-b306-458b-8b39-96deb3b575fa-plugins-config-volume\") pod \"payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn\" (UID: \"81b5ac0a-b306-458b-8b39-96deb3b575fa\") " pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" Nov 03 20:18:51.586292 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.586245 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" event={"ID":"a6b8b440-7a7c-4713-bce7-529cf543c29d","Type":"ContainerStarted","Data":"9950a33eb973d7f9a27b8fa158bc4e85656e66a1298f5bf280da89a65466943d"} Nov 03 20:18:51.604740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.604156 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6rpn\" (UniqueName: \"kubernetes.io/projected/81b5ac0a-b306-458b-8b39-96deb3b575fa-kube-api-access-k6rpn\") pod \"payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn\" (UID: \"81b5ac0a-b306-458b-8b39-96deb3b575fa\") " pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" Nov 03 20:18:51.738786 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:51.736691 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" Nov 03 20:18:52.237564 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:52.234689 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t"] Nov 03 20:18:52.386491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:52.369549 4457 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 20:18:52.386491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:52.369616 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 20:18:52.386491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:52.372484 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:18:52.422755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:52.422725 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn"] Nov 03 20:18:52.633654 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:52.633618 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" event={"ID":"81b5ac0a-b306-458b-8b39-96deb3b575fa","Type":"ContainerStarted","Data":"89904689beceaf5a1e9929e6cccc9d91eeddf5c65db67dfc70958b0600d72440"} Nov 03 20:18:52.633827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:52.633672 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" event={"ID":"81b5ac0a-b306-458b-8b39-96deb3b575fa","Type":"ContainerStarted","Data":"e638321011eb8c478ec0de5ee0538ce2a30de935c756553b538456929ba5ac3f"} Nov 03 20:18:52.634794 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:52.634775 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" Nov 03 20:18:52.637039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:52.636970 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" event={"ID":"c254c14f-13f1-4e20-bd27-14585726663f","Type":"ContainerStarted","Data":"8771c19020b4901bc0c96afa8c44d8dc8e7618713784fb46a9a038d75f81e6ad"} Nov 03 20:18:52.637039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:52.637028 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" event={"ID":"c254c14f-13f1-4e20-bd27-14585726663f","Type":"ContainerStarted","Data":"c276a343085f2594c755fbc93d2d3d93eeb5d69d5a7d366728f642dac5db23c1"} Nov 03 20:18:52.693282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:52.693206 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" podStartSLOduration=1.6931854899999998 podStartE2EDuration="1.69318549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:52.652620158 +0000 UTC m=+1311.028216932" watchObservedRunningTime="2026-11-03 20:18:52.69318549 +0000 UTC m=+1311.068782239" Nov 03 20:18:52.694398 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:52.694340 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" podStartSLOduration=1.694330462 podStartE2EDuration="1.694330462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:18:52.691036455 +0000 UTC m=+1311.066633218" watchObservedRunningTime="2026-11-03 20:18:52.694330462 +0000 UTC m=+1311.069927208" Nov 03 20:18:53.378411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:53.366987 4457 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 20:18:53.378411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:53.367099 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 20:18:53.646439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:53.646337 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" Nov 03 20:18:54.363818 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:54.363779 4457 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 20:18:54.364120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:54.364092 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 20:18:55.062758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:55.059476 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:18:55.363464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:55.361829 4457 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 20:18:55.363464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:55.361901 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 20:18:56.365843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:56.365803 4457 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 03 20:18:56.366481 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:56.366444 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 03 20:18:57.369510 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:57.369336 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:18:57.370881 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:18:57.370857 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:19:00.075424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:00.073433 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf"] Nov 03 20:19:00.139017 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:00.138966 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b223ef8a-rqvwf"] Nov 03 20:19:01.762592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:01.762549 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5c653a-d229-4ff8-b937-fdba3b5c7012" path="/var/lib/kubelet/pods/ab5c653a-d229-4ff8-b937-fdba3b5c7012/volumes" Nov 03 20:19:03.545740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:03.545709 4457 scope.go:122] "RemoveContainer" containerID="7c6187dec557345a8b20b75deb079bdc1f05a0468880bc4d503abc98c1a70599" Nov 03 20:19:03.851223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:03.851188 4457 scope.go:122] "RemoveContainer" containerID="ffa6fbf13c9c783ecc3107350c88f2bf5b85f4792a6708eff390c3c642159e88" Nov 03 20:19:04.136638 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:04.136245 4457 scope.go:122] "RemoveContainer" containerID="2814324eedcc19edc699eebf2c13a8b5b76b28b84aadc69e3af5cb52c92ef3e8" Nov 03 20:19:04.293413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:04.288333 4457 scope.go:122] "RemoveContainer" containerID="6fa19c3e2c8af374d3b8f3b0b776ad20003b4a36e444da7a972eb868c56c0809" Nov 03 20:19:04.313916 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:04.311775 4457 scope.go:122] "RemoveContainer" containerID="d44f70279df7fb1352959fe0563eccde3383166eb33ab6dd8434d4f4d0bde75d" Nov 03 20:19:04.462412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:04.437084 4457 scope.go:122] "RemoveContainer" containerID="82a1aaeca0666d4efb176777434b53953f4834a143ad186b2d8c7f6e8bc62d65" Nov 03 20:19:04.554597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:04.554555 4457 scope.go:122] "RemoveContainer" containerID="b9fbc7058e2d7ef4c99fdb4e43550f304ba51f1f24abed0c675465d49e4e0107" Nov 03 20:19:04.708465 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:04.700064 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" Nov 03 20:19:04.749417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:04.736548 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" Nov 03 20:19:04.751377 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:04.749777 4457 scope.go:122] "RemoveContainer" containerID="8e81450a683a2e8a52d3ee2e70665816879ea8968b7a2b395beca34360a256e7" Nov 03 20:19:05.341804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:05.340102 4457 scope.go:122] "RemoveContainer" containerID="94b49037c061ae1e64a0f7ee9119f6291c2e9899b530cdd594cbcf10d98fff64" Nov 03 20:19:06.152118 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:19:06.152074 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:19:15.047909 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.046754 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc"] Nov 03 20:19:15.143097 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.143064 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc"] Nov 03 20:19:15.165593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.151564 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.184419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.179380 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9af0cd-openshift-default-dockercfg-plvgt\"" Nov 03 20:19:15.632687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.626756 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c79f72-431b-4137-950f-b5f8d68f9a9e-istiod-ca-cert\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.632687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.626834 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-certs\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.632687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.626954 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-podinfo\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.632687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.627014 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-socket\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.632687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.627087 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-token\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.632687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.627131 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-credential-socket\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.632687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.627361 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxq4c\" (UniqueName: \"kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-kube-api-access-wxq4c\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.632687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.627434 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-envoy\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.632687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.627472 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-data\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.729784 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.729664 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxq4c\" (UniqueName: \"kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-kube-api-access-wxq4c\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.729784 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.729764 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-envoy\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.729998 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.729871 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-data\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.730068 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.730049 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c79f72-431b-4137-950f-b5f8d68f9a9e-istiod-ca-cert\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.730142 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.730117 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-certs\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.730226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.730213 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-podinfo\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.730272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.730255 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-socket\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.730315 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.730301 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-token\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.730355 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.730329 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-credential-socket\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.730773 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.730758 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-credential-socket\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.731618 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.731591 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-certs\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.731788 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.731639 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-data\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.731908 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.731884 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-socket\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.732212 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.732186 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c79f72-431b-4137-950f-b5f8d68f9a9e-istiod-ca-cert\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.733824 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.733795 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-envoy\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.738490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.734549 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-podinfo\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.751410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.749892 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxq4c\" (UniqueName: \"kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-kube-api-access-wxq4c\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.751410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.750535 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-token\") pod \"e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:15.943668 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:15.943627 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:16.322772 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:16.322011 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc"] Nov 03 20:19:16.346267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:16.346221 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:16.346531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:16.346502 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:16.346665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:16.346562 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:16.753820 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:16.753729 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" event={"ID":"09c79f72-431b-4137-950f-b5f8d68f9a9e","Type":"ContainerStarted","Data":"94d8811fd536c2ba120cffa921554030600eeed3a885381a00c87370091c9fe1"} Nov 03 20:19:16.753820 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:16.753802 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" event={"ID":"09c79f72-431b-4137-950f-b5f8d68f9a9e","Type":"ContainerStarted","Data":"a1c8ad50b069bd6dfa9cde96ee1bcfeeae4c68f8f5cbfe41e03fcccd5cc446c4"} Nov 03 20:19:16.790412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:16.789635 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" podStartSLOduration=1.789616649 podStartE2EDuration="1.789616649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:16.782437232 +0000 UTC m=+1335.158033983" watchObservedRunningTime="2026-11-03 20:19:16.789616649 +0000 UTC m=+1335.165213396" Nov 03 20:19:16.956423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:16.954545 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:17.951889 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:17.951857 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:18.797811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:18.797769 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:18.798977 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:18.798951 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:19:20.259039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:20.259000 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz"] Nov 03 20:19:20.263965 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:20.263940 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" Nov 03 20:19:20.271852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:20.271822 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9af0cd\"" Nov 03 20:19:20.273591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:20.273564 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9af0cd-dockercfg-bppt7\"" Nov 03 20:19:20.281911 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:20.281886 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz"] Nov 03 20:19:20.354731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:20.354672 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz\" (UID: \"d1acef2b-8495-41ff-8a07-1797e7a8a1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" Nov 03 20:19:20.369591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:20.355284 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7jsq\" (UniqueName: \"kubernetes.io/projected/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-kube-api-access-r7jsq\") pod \"payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz\" (UID: \"d1acef2b-8495-41ff-8a07-1797e7a8a1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" Nov 03 20:19:20.511631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:20.511536 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7jsq\" (UniqueName: \"kubernetes.io/projected/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-kube-api-access-r7jsq\") pod \"payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz\" (UID: \"d1acef2b-8495-41ff-8a07-1797e7a8a1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" Nov 03 20:19:20.512061 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:20.512036 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz\" (UID: \"d1acef2b-8495-41ff-8a07-1797e7a8a1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" Nov 03 20:19:20.517409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:20.513357 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz\" (UID: \"d1acef2b-8495-41ff-8a07-1797e7a8a1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" Nov 03 20:19:20.582049 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:20.580961 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7jsq\" (UniqueName: \"kubernetes.io/projected/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-kube-api-access-r7jsq\") pod \"payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz\" (UID: \"d1acef2b-8495-41ff-8a07-1797e7a8a1a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" Nov 03 20:19:20.601247 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:20.599681 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" Nov 03 20:19:21.579723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:21.579696 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz"] Nov 03 20:19:21.580136 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:19:21.579821 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1acef2b_8495_41ff_8a07_1797e7a8a1a6.slice/crio-8d49ed7796074448ca8c7be157e8808b215b01a3c3fb8349443ffac86374233e WatchSource:0}: Error finding container 8d49ed7796074448ca8c7be157e8808b215b01a3c3fb8349443ffac86374233e: Status 404 returned error can't find the container with id 8d49ed7796074448ca8c7be157e8808b215b01a3c3fb8349443ffac86374233e Nov 03 20:19:21.906603 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:21.906521 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" event={"ID":"d1acef2b-8495-41ff-8a07-1797e7a8a1a6","Type":"ContainerStarted","Data":"d9ccab01694976ba56bb7474aa9c6ce4acd8b3daeac836ad15f66e1a2e6f9f5f"} Nov 03 20:19:21.906603 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:21.906577 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" event={"ID":"d1acef2b-8495-41ff-8a07-1797e7a8a1a6","Type":"ContainerStarted","Data":"8d49ed7796074448ca8c7be157e8808b215b01a3c3fb8349443ffac86374233e"} Nov 03 20:19:21.906796 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:21.906690 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" Nov 03 20:19:21.929348 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:21.929285 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" podStartSLOduration=1.929269761 podStartE2EDuration="1.929269761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:21.924125281 +0000 UTC m=+1340.299722029" watchObservedRunningTime="2026-11-03 20:19:21.929269761 +0000 UTC m=+1340.304866513" Nov 03 20:19:25.885433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:25.885288 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh"] Nov 03 20:19:25.901088 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:25.892903 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" Nov 03 20:19:25.904951 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:25.904915 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh"] Nov 03 20:19:25.963779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:25.963751 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-3ef4b9\"" Nov 03 20:19:25.964112 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:25.964090 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-3ef4b9-dockercfg-jcmkq\"" Nov 03 20:19:26.087540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:26.077293 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqqd6\" (UniqueName: \"kubernetes.io/projected/61c89239-0110-425d-9aff-2652c9b8dfd7-kube-api-access-cqqd6\") pod \"payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh\" (UID: \"61c89239-0110-425d-9aff-2652c9b8dfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" Nov 03 20:19:26.087908 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:26.087882 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c89239-0110-425d-9aff-2652c9b8dfd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh\" (UID: \"61c89239-0110-425d-9aff-2652c9b8dfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" Nov 03 20:19:26.190666 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:26.190561 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqqd6\" (UniqueName: \"kubernetes.io/projected/61c89239-0110-425d-9aff-2652c9b8dfd7-kube-api-access-cqqd6\") pod \"payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh\" (UID: \"61c89239-0110-425d-9aff-2652c9b8dfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" Nov 03 20:19:26.190666 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:26.190659 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c89239-0110-425d-9aff-2652c9b8dfd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh\" (UID: \"61c89239-0110-425d-9aff-2652c9b8dfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" Nov 03 20:19:26.194417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:26.191956 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c89239-0110-425d-9aff-2652c9b8dfd7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh\" (UID: \"61c89239-0110-425d-9aff-2652c9b8dfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" Nov 03 20:19:26.231410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:26.231187 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqqd6\" (UniqueName: \"kubernetes.io/projected/61c89239-0110-425d-9aff-2652c9b8dfd7-kube-api-access-cqqd6\") pod \"payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh\" (UID: \"61c89239-0110-425d-9aff-2652c9b8dfd7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" Nov 03 20:19:26.520063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:26.519969 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" Nov 03 20:19:26.824545 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:19:26.824502 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c89239_0110_425d_9aff_2652c9b8dfd7.slice/crio-577fd7d1b16d7495d786cde00558545906c339f0771e0aca38b8bf3432bcc779 WatchSource:0}: Error finding container 577fd7d1b16d7495d786cde00558545906c339f0771e0aca38b8bf3432bcc779: Status 404 returned error can't find the container with id 577fd7d1b16d7495d786cde00558545906c339f0771e0aca38b8bf3432bcc779 Nov 03 20:19:26.825996 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:26.825965 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh"] Nov 03 20:19:26.963709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:26.963673 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" event={"ID":"61c89239-0110-425d-9aff-2652c9b8dfd7","Type":"ContainerStarted","Data":"60739149f7f2a5dae5934897651e352c66ebb2628949eeb43852f81f29093c0b"} Nov 03 20:19:26.964242 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:26.963734 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" event={"ID":"61c89239-0110-425d-9aff-2652c9b8dfd7","Type":"ContainerStarted","Data":"577fd7d1b16d7495d786cde00558545906c339f0771e0aca38b8bf3432bcc779"} Nov 03 20:19:26.964242 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:26.963789 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" Nov 03 20:19:26.985713 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:26.985658 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" podStartSLOduration=1.9856450909999999 podStartE2EDuration="1.985645091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:26.981468397 +0000 UTC m=+1345.357065146" watchObservedRunningTime="2026-11-03 20:19:26.985645091 +0000 UTC m=+1345.361241862" Nov 03 20:19:30.033473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:30.033436 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp"] Nov 03 20:19:30.049441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:30.042152 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3293116-xjpbp"] Nov 03 20:19:31.920184 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:31.920147 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a45915a-881d-40e3-bc6f-aa9c2f03ad06" path="/var/lib/kubelet/pods/6a45915a-881d-40e3-bc6f-aa9c2f03ad06/volumes" Nov 03 20:19:32.919958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:32.919922 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" Nov 03 20:19:36.159957 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:19:36.159898 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:19:36.871678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:36.871643 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws"] Nov 03 20:19:36.877357 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:36.877329 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" Nov 03 20:19:36.890772 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:36.889574 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-06294c6a\"" Nov 03 20:19:36.890772 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:36.890596 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws"] Nov 03 20:19:36.913152 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:36.912974 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-06294c6a-dockercfg-6q7k2\"" Nov 03 20:19:37.034665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.034622 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w76wx\" (UniqueName: \"kubernetes.io/projected/2f09c9a5-5bfe-4078-912e-48b2569e2d76-kube-api-access-w76wx\") pod \"payload-processing-e2e-mt-06294c6a-558877d94b-v8bws\" (UID: \"2f09c9a5-5bfe-4078-912e-48b2569e2d76\") " pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" Nov 03 20:19:37.036674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.036641 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f09c9a5-5bfe-4078-912e-48b2569e2d76-plugins-config-volume\") pod \"payload-processing-e2e-mt-06294c6a-558877d94b-v8bws\" (UID: \"2f09c9a5-5bfe-4078-912e-48b2569e2d76\") " pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" Nov 03 20:19:37.165540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.156565 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f09c9a5-5bfe-4078-912e-48b2569e2d76-plugins-config-volume\") pod \"payload-processing-e2e-mt-06294c6a-558877d94b-v8bws\" (UID: \"2f09c9a5-5bfe-4078-912e-48b2569e2d76\") " pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" Nov 03 20:19:37.165540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.156768 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w76wx\" (UniqueName: \"kubernetes.io/projected/2f09c9a5-5bfe-4078-912e-48b2569e2d76-kube-api-access-w76wx\") pod \"payload-processing-e2e-mt-06294c6a-558877d94b-v8bws\" (UID: \"2f09c9a5-5bfe-4078-912e-48b2569e2d76\") " pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" Nov 03 20:19:37.165540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.157887 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f09c9a5-5bfe-4078-912e-48b2569e2d76-plugins-config-volume\") pod \"payload-processing-e2e-mt-06294c6a-558877d94b-v8bws\" (UID: \"2f09c9a5-5bfe-4078-912e-48b2569e2d76\") " pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" Nov 03 20:19:37.226409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.223733 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w76wx\" (UniqueName: \"kubernetes.io/projected/2f09c9a5-5bfe-4078-912e-48b2569e2d76-kube-api-access-w76wx\") pod \"payload-processing-e2e-mt-06294c6a-558877d94b-v8bws\" (UID: \"2f09c9a5-5bfe-4078-912e-48b2569e2d76\") " pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" Nov 03 20:19:37.306406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.300414 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z"] Nov 03 20:19:37.429485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.426124 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z"] Nov 03 20:19:37.429485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.426569 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.432325 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.432304 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-3ef4b9-serving-cert\"" Nov 03 20:19:37.504440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.504251 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" Nov 03 20:19:37.574797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.573339 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.574797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.573450 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.574797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.573569 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.574797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.573860 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-976g7\" (UniqueName: \"kubernetes.io/projected/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-kube-api-access-976g7\") pod \"maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.687459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.681048 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.687459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.681340 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-976g7\" (UniqueName: \"kubernetes.io/projected/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-kube-api-access-976g7\") pod \"maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.687459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.681492 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.687459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.681526 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.687459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.682713 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.687459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.687280 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.694614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.694509 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.711026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.701792 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-976g7\" (UniqueName: \"kubernetes.io/projected/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-kube-api-access-976g7\") pod \"maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.764411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.757904 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:37.850539 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:37.849483 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws"] Nov 03 20:19:38.217461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:38.207474 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" Nov 03 20:19:38.273851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:38.273810 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" event={"ID":"2f09c9a5-5bfe-4078-912e-48b2569e2d76","Type":"ContainerStarted","Data":"ec1669f8e3b97b01c4ecaeef05a52a9d71cbbf89cbfdff672967911bf28893e4"} Nov 03 20:19:38.357841 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:38.357808 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z"] Nov 03 20:19:38.380433 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:19:38.379328 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cd7625_f8fb_42aa_a39a_5ea33b4b8831.slice/crio-36a773ec3987408d1b02d00b0327b1fe455b2bbec247df2add40da736d7a6bae WatchSource:0}: Error finding container 36a773ec3987408d1b02d00b0327b1fe455b2bbec247df2add40da736d7a6bae: Status 404 returned error can't find the container with id 36a773ec3987408d1b02d00b0327b1fe455b2bbec247df2add40da736d7a6bae Nov 03 20:19:39.286591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:39.286495 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" event={"ID":"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831","Type":"ContainerStarted","Data":"74a64a3c1a06a597a4750e476d28352ea1e4b32e2b94717ea73edb5b0cb01bab"} Nov 03 20:19:39.286591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:39.286549 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" event={"ID":"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831","Type":"ContainerStarted","Data":"36a773ec3987408d1b02d00b0327b1fe455b2bbec247df2add40da736d7a6bae"} Nov 03 20:19:39.287080 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:39.286604 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:39.288663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:39.288635 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" event={"ID":"2f09c9a5-5bfe-4078-912e-48b2569e2d76","Type":"ContainerStarted","Data":"967a718ba0402b60603d427fc948ef2b641f124b8a6568d94376fd0e0b632271"} Nov 03 20:19:39.288813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:39.288699 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" Nov 03 20:19:39.312569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:39.312520 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" podStartSLOduration=1.866684963 podStartE2EDuration="2.312507462s" totalImagesPullingTime="445.822499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:39.306309866 +0000 UTC m=+1357.681906614" watchObservedRunningTime="2026-11-03 20:19:39.312507462 +0000 UTC m=+1357.688104256" Nov 03 20:19:39.325306 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:39.325249 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" podStartSLOduration=3.325237943 podStartE2EDuration="3.325237943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:19:39.324608043 +0000 UTC m=+1357.700204790" watchObservedRunningTime="2026-11-03 20:19:39.325237943 +0000 UTC m=+1357.700834690" Nov 03 20:19:43.069064 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:43.068613 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:19:43.132769 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:43.131464 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:19:43.189413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:43.179501 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.038413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.037479 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.038413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.037765 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Nov 03 20:19:45.051409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.043460 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.051409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.043813 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:19:45.051409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.044094 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:19:45.076413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.059464 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.077536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.077492 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.115524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.115474 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 03 20:19:45.116950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.116921 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:19:45.139679 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.139638 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.158201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.158162 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.158649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.158620 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.172055 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.172018 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.206517 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.206469 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.373413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.371584 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:19:45.705251 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.705147 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.770421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.763442 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.842418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.841539 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:45.869562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:45.869305 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded" Nov 03 20:19:46.050689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.050644 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:19:46.086058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.086022 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:19:46.089342 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.088550 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:46.090144 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.090123 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:19:46.103414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.102480 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:46.103414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.102958 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:19:46.132681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.132634 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:46.133027 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.132994 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 03 20:19:46.133585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.133562 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:19:46.133704 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.133629 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:19:46.147410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.141471 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:46.147410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.142535 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:19:46.212585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.212540 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:46.224452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.215347 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:19:46.543489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.523633 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:19:46.762749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.762721 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:19:46.763155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.763137 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:19:46.775137 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.775090 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:46.777107 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.777084 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:19:46.844728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.841789 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:46.846295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.846269 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:19:46.895239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:46.895207 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:19:47.054682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:47.054632 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:47.056344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:47.055908 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:47.118667 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:47.118587 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:19:47.119129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:47.119088 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:19:47.132504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:47.131984 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:19:47.140995 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:47.140429 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:47.161415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:47.156676 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:19:47.161415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:47.157903 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:19:47.232130 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:47.232096 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:19:47.245573 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:47.245546 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:19:47.352264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:47.352236 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:19:50.321755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:50.321722 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" Nov 03 20:19:52.997299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:52.997261 4457 generic.go:356] "Generic (PLEG): container finished" podID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerID="9997f4ec20b8ebad6e4d59ad71e3d378339d5ca88f1b29495739890a9a50a094" exitCode=0 Nov 03 20:19:52.998498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:52.998466 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" event={"ID":"90af0b2d-03b0-4532-85ee-f3282cc9637a","Type":"ContainerDied","Data":"9997f4ec20b8ebad6e4d59ad71e3d378339d5ca88f1b29495739890a9a50a094"} Nov 03 20:19:52.999236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:52.999215 4457 scope.go:122] "RemoveContainer" containerID="9997f4ec20b8ebad6e4d59ad71e3d378339d5ca88f1b29495739890a9a50a094" Nov 03 20:19:53.002630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:53.002601 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:53.003200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:53.003174 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:53.003365 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:53.003347 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:19:54.038493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:54.038454 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" event={"ID":"90af0b2d-03b0-4532-85ee-f3282cc9637a","Type":"ContainerStarted","Data":"5ca8587907e5d38822cc1e455d1dc14cf611964d6deebcd6c87a8252100b28e2"} Nov 03 20:19:54.165736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:54.165703 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:19:54.166024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:54.165828 4457 patch_prober.go:29] interesting pod/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:19:54.166096 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:54.166078 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 20:19:55.165598 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:55.165562 4457 patch_prober.go:29] interesting pod/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:19:55.166205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:55.166172 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 20:19:56.164410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:56.164233 4457 patch_prober.go:29] interesting pod/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:19:56.164410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:56.164306 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 20:19:57.164820 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:57.164782 4457 patch_prober.go:29] interesting pod/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:19:57.165829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:57.165793 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 20:19:58.171433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:58.170544 4457 patch_prober.go:29] interesting pod/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:19:58.171433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:58.170615 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 20:19:59.167434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:59.165744 4457 patch_prober.go:29] interesting pod/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 03 20:19:59.167434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:19:59.165823 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 03 20:20:00.173404 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:00.173022 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:20:00.174498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:00.174472 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:20:00.175813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:00.175790 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:20:01.030852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:01.030813 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7"] Nov 03 20:20:01.037307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:01.037278 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2e6911f-hwfm7"] Nov 03 20:20:01.765085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:01.765053 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b94fdcc8-e4e1-4384-beb7-aad82f1bed5b" path="/var/lib/kubelet/pods/b94fdcc8-e4e1-4384-beb7-aad82f1bed5b/volumes" Nov 03 20:20:05.094487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:05.094421 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn"] Nov 03 20:20:05.109357 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:05.109293 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt"] Nov 03 20:20:05.119212 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:05.119168 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06abad0c-9vzzn"] Nov 03 20:20:05.127909 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:05.127875 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8a81f6-wlwpt"] Nov 03 20:20:05.575713 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:05.575678 4457 scope.go:122] "RemoveContainer" containerID="caa54df55478235b92fb31225db644f995a102d60225ad4424e6422f01a337b7" Nov 03 20:20:05.760343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:05.760307 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17465221-8104-4091-8e04-0e89431e264d" path="/var/lib/kubelet/pods/17465221-8104-4091-8e04-0e89431e264d/volumes" Nov 03 20:20:05.761230 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:05.761205 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81f782b7-a701-419b-9326-7bc84605807f" path="/var/lib/kubelet/pods/81f782b7-a701-419b-9326-7bc84605807f/volumes" Nov 03 20:20:06.151849 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:20:06.151810 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:20:07.500273 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.500233 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b"] Nov 03 20:20:07.508431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.508405 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" Nov 03 20:20:07.514878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.514851 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b"] Nov 03 20:20:07.519655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.519611 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ba8f5b\"" Nov 03 20:20:07.544662 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.544625 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ba8f5b-dockercfg-q4smf\"" Nov 03 20:20:07.589989 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.589956 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7"] Nov 03 20:20:07.598253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.598226 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" Nov 03 20:20:07.599676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.598815 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7"] Nov 03 20:20:07.661414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.639174 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ffz7\" (UniqueName: \"kubernetes.io/projected/1a6ffae7-5a36-4221-8263-e56842c7d0ea-kube-api-access-7ffz7\") pod \"payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b\" (UID: \"1a6ffae7-5a36-4221-8263-e56842c7d0ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" Nov 03 20:20:07.661414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.639661 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6ffae7-5a36-4221-8263-e56842c7d0ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b\" (UID: \"1a6ffae7-5a36-4221-8263-e56842c7d0ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" Nov 03 20:20:07.842455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.838789 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz9q9\" (UniqueName: \"kubernetes.io/projected/ead54f07-35d4-49da-b436-54f0421b6261-kube-api-access-nz9q9\") pod \"payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7\" (UID: \"ead54f07-35d4-49da-b436-54f0421b6261\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" Nov 03 20:20:07.842455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.838941 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6ffae7-5a36-4221-8263-e56842c7d0ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b\" (UID: \"1a6ffae7-5a36-4221-8263-e56842c7d0ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" Nov 03 20:20:07.842455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.839118 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ffz7\" (UniqueName: \"kubernetes.io/projected/1a6ffae7-5a36-4221-8263-e56842c7d0ea-kube-api-access-7ffz7\") pod \"payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b\" (UID: \"1a6ffae7-5a36-4221-8263-e56842c7d0ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" Nov 03 20:20:07.842455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.839166 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead54f07-35d4-49da-b436-54f0421b6261-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7\" (UID: \"ead54f07-35d4-49da-b436-54f0421b6261\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" Nov 03 20:20:07.874440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.847747 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6ffae7-5a36-4221-8263-e56842c7d0ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b\" (UID: \"1a6ffae7-5a36-4221-8263-e56842c7d0ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" Nov 03 20:20:07.925949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.907764 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ffz7\" (UniqueName: \"kubernetes.io/projected/1a6ffae7-5a36-4221-8263-e56842c7d0ea-kube-api-access-7ffz7\") pod \"payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b\" (UID: \"1a6ffae7-5a36-4221-8263-e56842c7d0ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" Nov 03 20:20:07.960477 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.959610 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead54f07-35d4-49da-b436-54f0421b6261-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7\" (UID: \"ead54f07-35d4-49da-b436-54f0421b6261\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" Nov 03 20:20:07.960477 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.959892 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz9q9\" (UniqueName: \"kubernetes.io/projected/ead54f07-35d4-49da-b436-54f0421b6261-kube-api-access-nz9q9\") pod \"payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7\" (UID: \"ead54f07-35d4-49da-b436-54f0421b6261\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" Nov 03 20:20:07.963244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.961289 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead54f07-35d4-49da-b436-54f0421b6261-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7\" (UID: \"ead54f07-35d4-49da-b436-54f0421b6261\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" Nov 03 20:20:07.972943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.972549 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz9q9\" (UniqueName: \"kubernetes.io/projected/ead54f07-35d4-49da-b436-54f0421b6261-kube-api-access-nz9q9\") pod \"payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7\" (UID: \"ead54f07-35d4-49da-b436-54f0421b6261\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" Nov 03 20:20:07.995655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:07.990679 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" Nov 03 20:20:08.134489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:08.134079 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" Nov 03 20:20:08.315735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:08.315704 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7"] Nov 03 20:20:08.408432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:08.408372 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" event={"ID":"ead54f07-35d4-49da-b436-54f0421b6261","Type":"ContainerStarted","Data":"41c14940433d0f58680a6e1fefa8aee091fc8e28404053bcc65282e02d1a48c2"} Nov 03 20:20:08.488568 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:20:08.488523 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6ffae7_5a36_4221_8263_e56842c7d0ea.slice/crio-8d2b60a792e4fbec0a1ca9b5004a9333126f3cadf733c0c3de0570b6f34bd492 WatchSource:0}: Error finding container 8d2b60a792e4fbec0a1ca9b5004a9333126f3cadf733c0c3de0570b6f34bd492: Status 404 returned error can't find the container with id 8d2b60a792e4fbec0a1ca9b5004a9333126f3cadf733c0c3de0570b6f34bd492 Nov 03 20:20:08.494485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:08.493207 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b"] Nov 03 20:20:09.419962 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:09.419921 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" event={"ID":"ead54f07-35d4-49da-b436-54f0421b6261","Type":"ContainerStarted","Data":"c1c61ed92b2a4326fd297251a2475e7b97cb0b7e89701addfa4ea01d65fd837b"} Nov 03 20:20:09.420406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:09.420022 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" Nov 03 20:20:09.421926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:09.421901 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" event={"ID":"1a6ffae7-5a36-4221-8263-e56842c7d0ea","Type":"ContainerStarted","Data":"26483bd49bc66771e0aadeea8d84d3236fae149edb9739c7e6b12468eb692804"} Nov 03 20:20:09.422038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:09.421943 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" event={"ID":"1a6ffae7-5a36-4221-8263-e56842c7d0ea","Type":"ContainerStarted","Data":"8d2b60a792e4fbec0a1ca9b5004a9333126f3cadf733c0c3de0570b6f34bd492"} Nov 03 20:20:09.422038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:09.421988 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" Nov 03 20:20:09.441963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:09.441917 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" podStartSLOduration=2.441904657 podStartE2EDuration="2.441904657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:09.437929025 +0000 UTC m=+1387.813525774" watchObservedRunningTime="2026-11-03 20:20:09.441904657 +0000 UTC m=+1387.817501400" Nov 03 20:20:09.459227 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:09.459171 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" podStartSLOduration=2.459159033 podStartE2EDuration="2.459159033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:09.455580285 +0000 UTC m=+1387.831177039" watchObservedRunningTime="2026-11-03 20:20:09.459159033 +0000 UTC m=+1387.834755801" Nov 03 20:20:10.051231 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:10.051198 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l"] Nov 03 20:20:10.056551 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:10.056517 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65ae418c-6xl2l"] Nov 03 20:20:11.767137 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:11.767098 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acab3741-a3ca-415d-b8e4-97143771efb9" path="/var/lib/kubelet/pods/acab3741-a3ca-415d-b8e4-97143771efb9/volumes" Nov 03 20:20:13.027552 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.027511 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md"] Nov 03 20:20:13.032636 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.032612 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baa0a88a-pv8md"] Nov 03 20:20:13.222173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.222136 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d"] Nov 03 20:20:13.227963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.227940 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" Nov 03 20:20:13.233233 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.233197 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d"] Nov 03 20:20:13.233726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.233702 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:20:13.234931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.234912 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:20:13.311792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.311761 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d\" (UID: \"06817c43-6630-44cb-ad56-f5c4a8e04f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" Nov 03 20:20:13.312004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.311978 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6cb\" (UniqueName: \"kubernetes.io/projected/06817c43-6630-44cb-ad56-f5c4a8e04f25-kube-api-access-9d6cb\") pod \"maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d\" (UID: \"06817c43-6630-44cb-ad56-f5c4a8e04f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" Nov 03 20:20:13.312179 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.312159 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d\" (UID: \"06817c43-6630-44cb-ad56-f5c4a8e04f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" Nov 03 20:20:13.414307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.414254 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6cb\" (UniqueName: \"kubernetes.io/projected/06817c43-6630-44cb-ad56-f5c4a8e04f25-kube-api-access-9d6cb\") pod \"maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d\" (UID: \"06817c43-6630-44cb-ad56-f5c4a8e04f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" Nov 03 20:20:13.414535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.414517 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d\" (UID: \"06817c43-6630-44cb-ad56-f5c4a8e04f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" Nov 03 20:20:13.414746 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.414727 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d\" (UID: \"06817c43-6630-44cb-ad56-f5c4a8e04f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" Nov 03 20:20:13.415325 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.415261 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d\" (UID: \"06817c43-6630-44cb-ad56-f5c4a8e04f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" Nov 03 20:20:13.417237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.417212 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d\" (UID: \"06817c43-6630-44cb-ad56-f5c4a8e04f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" Nov 03 20:20:13.426952 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.426922 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6cb\" (UniqueName: \"kubernetes.io/projected/06817c43-6630-44cb-ad56-f5c4a8e04f25-kube-api-access-9d6cb\") pod \"maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d\" (UID: \"06817c43-6630-44cb-ad56-f5c4a8e04f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" Nov 03 20:20:13.579949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.579874 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" Nov 03 20:20:13.777943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.777899 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f216dffd-5ec3-4ec9-8c40-9534f69ee329" path="/var/lib/kubelet/pods/f216dffd-5ec3-4ec9-8c40-9534f69ee329/volumes" Nov 03 20:20:13.880538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:13.880498 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d"] Nov 03 20:20:14.516236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:14.516138 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" event={"ID":"06817c43-6630-44cb-ad56-f5c4a8e04f25","Type":"ContainerStarted","Data":"b40e7ac2c90415020d0daf3d9d3969ac94b25fe60e538392e3ed21499cd2a55d"} Nov 03 20:20:14.516236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:14.516202 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" event={"ID":"06817c43-6630-44cb-ad56-f5c4a8e04f25","Type":"ContainerStarted","Data":"4d25b6fb6b2acf9ec5cde50b29347940c71202a4ae37712460f7846602524d51"} Nov 03 20:20:14.535622 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:14.535566 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" podStartSLOduration=1.535551453 podStartE2EDuration="1.535551453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:14.534183511 +0000 UTC m=+1392.909780259" watchObservedRunningTime="2026-11-03 20:20:14.535551453 +0000 UTC m=+1392.911148202" Nov 03 20:20:15.530841 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:15.530742 4457 generic.go:356] "Generic (PLEG): container finished" podID="06817c43-6630-44cb-ad56-f5c4a8e04f25" containerID="b40e7ac2c90415020d0daf3d9d3969ac94b25fe60e538392e3ed21499cd2a55d" exitCode=0 Nov 03 20:20:15.531300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:15.531072 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" event={"ID":"06817c43-6630-44cb-ad56-f5c4a8e04f25","Type":"ContainerDied","Data":"b40e7ac2c90415020d0daf3d9d3969ac94b25fe60e538392e3ed21499cd2a55d"} Nov 03 20:20:16.747719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:16.747691 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" Nov 03 20:20:16.786866 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:16.786829 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06817c43-6630-44cb-ad56-f5c4a8e04f25-kube-api-access-9d6cb\" (UniqueName: \"kubernetes.io/projected/06817c43-6630-44cb-ad56-f5c4a8e04f25-kube-api-access-9d6cb\") pod \"06817c43-6630-44cb-ad56-f5c4a8e04f25\" (UID: \"06817c43-6630-44cb-ad56-f5c4a8e04f25\") " Nov 03 20:20:16.787475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:16.787152 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-xks-ca\") pod \"06817c43-6630-44cb-ad56-f5c4a8e04f25\" (UID: \"06817c43-6630-44cb-ad56-f5c4a8e04f25\") " Nov 03 20:20:16.787475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:16.787261 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-openshift-ca\") pod \"06817c43-6630-44cb-ad56-f5c4a8e04f25\" (UID: \"06817c43-6630-44cb-ad56-f5c4a8e04f25\") " Nov 03 20:20:16.788062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:16.788037 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-openshift-ca" pod "06817c43-6630-44cb-ad56-f5c4a8e04f25" (UID: "06817c43-6630-44cb-ad56-f5c4a8e04f25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:16.789833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:16.789801 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06817c43-6630-44cb-ad56-f5c4a8e04f25-kube-api-access-9d6cb" pod "06817c43-6630-44cb-ad56-f5c4a8e04f25" (UID: "06817c43-6630-44cb-ad56-f5c4a8e04f25"). InnerVolumeSpecName "kube-api-access-9d6cb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:16.809797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:16.809765 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-xks-ca" pod "06817c43-6630-44cb-ad56-f5c4a8e04f25" (UID: "06817c43-6630-44cb-ad56-f5c4a8e04f25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:16.890119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:16.890079 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:16.890119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:16.890115 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06817c43-6630-44cb-ad56-f5c4a8e04f25-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:16.890119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:16.890125 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d6cb\" (UniqueName: \"kubernetes.io/projected/06817c43-6630-44cb-ad56-f5c4a8e04f25-kube-api-access-9d6cb\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:17.551914 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:17.551887 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" Nov 03 20:20:17.552079 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:17.551916 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d" event={"ID":"06817c43-6630-44cb-ad56-f5c4a8e04f25","Type":"ContainerDied","Data":"4d25b6fb6b2acf9ec5cde50b29347940c71202a4ae37712460f7846602524d51"} Nov 03 20:20:17.552079 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:17.551965 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d25b6fb6b2acf9ec5cde50b29347940c71202a4ae37712460f7846602524d51" Nov 03 20:20:18.048883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:18.048844 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk"] Nov 03 20:20:18.067452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:18.067405 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87c167e-sj4bk"] Nov 03 20:20:19.840021 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:19.839972 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00169716-105a-4f81-a435-bf13cfb20a8d" path="/var/lib/kubelet/pods/00169716-105a-4f81-a435-bf13cfb20a8d/volumes" Nov 03 20:20:20.106541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:20.106448 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp"] Nov 03 20:20:20.195747 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:20.195699 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7069a6d9-p2slp"] Nov 03 20:20:20.435893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:20.435805 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" Nov 03 20:20:20.436132 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:20.436108 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" Nov 03 20:20:21.108515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:21.079733 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk"] Nov 03 20:20:21.109023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:21.108773 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c89d401-xrmkk"] Nov 03 20:20:21.776193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:21.776157 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6fa0c0d-8a71-4ffa-ac05-04653af8b63c" path="/var/lib/kubelet/pods/c6fa0c0d-8a71-4ffa-ac05-04653af8b63c/volumes" Nov 03 20:20:21.777295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:21.777269 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81ab721-84a4-4df3-bcde-91a7b2c7cc8d" path="/var/lib/kubelet/pods/e81ab721-84a4-4df3-bcde-91a7b2c7cc8d/volumes" Nov 03 20:20:27.605987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.605948 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf"] Nov 03 20:20:27.606943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.606920 4457 cpu_manager.go:414] "containerMap: removing container" podUID="06817c43-6630-44cb-ad56-f5c4a8e04f25" containerName="revoke-keys" Nov 03 20:20:27.607067 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.606944 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06817c43-6630-44cb-ad56-f5c4a8e04f25" containerName="revoke-keys" Nov 03 20:20:27.607250 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.607234 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="06817c43-6630-44cb-ad56-f5c4a8e04f25" containerName="revoke-keys" Nov 03 20:20:27.611442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.611377 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" Nov 03 20:20:27.620827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.620804 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf"] Nov 03 20:20:27.656509 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.655955 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:20:27.656911 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.656882 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:20:27.772457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.772420 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcm46\" (UniqueName: \"kubernetes.io/projected/c6ab47a3-5965-48cb-b127-4c927ceaf51a-kube-api-access-qcm46\") pod \"maas-api-revoke-sub-d278d729-2mdhf\" (UID: \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" Nov 03 20:20:27.772783 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.772747 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d278d729-2mdhf\" (UID: \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" Nov 03 20:20:27.772913 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.772847 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d278d729-2mdhf\" (UID: \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" Nov 03 20:20:27.875497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.875374 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcm46\" (UniqueName: \"kubernetes.io/projected/c6ab47a3-5965-48cb-b127-4c927ceaf51a-kube-api-access-qcm46\") pod \"maas-api-revoke-sub-d278d729-2mdhf\" (UID: \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" Nov 03 20:20:27.875497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.875468 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d278d729-2mdhf\" (UID: \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" Nov 03 20:20:27.875724 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.875505 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d278d729-2mdhf\" (UID: \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" Nov 03 20:20:27.876275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.876240 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d278d729-2mdhf\" (UID: \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" Nov 03 20:20:27.878050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.878026 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d278d729-2mdhf\" (UID: \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" Nov 03 20:20:27.888104 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.888079 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcm46\" (UniqueName: \"kubernetes.io/projected/c6ab47a3-5965-48cb-b127-4c927ceaf51a-kube-api-access-qcm46\") pod \"maas-api-revoke-sub-d278d729-2mdhf\" (UID: \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" Nov 03 20:20:27.929322 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:27.929291 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" Nov 03 20:20:28.222088 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:28.222054 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf"] Nov 03 20:20:28.222478 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:20:28.222454 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ab47a3_5965_48cb_b127_4c927ceaf51a.slice/crio-669713e288a0ce610563f75230e6fd23addb8c042ba73ae44cac3da22bb6677a WatchSource:0}: Error finding container 669713e288a0ce610563f75230e6fd23addb8c042ba73ae44cac3da22bb6677a: Status 404 returned error can't find the container with id 669713e288a0ce610563f75230e6fd23addb8c042ba73ae44cac3da22bb6677a Nov 03 20:20:28.720251 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:28.720219 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" event={"ID":"c6ab47a3-5965-48cb-b127-4c927ceaf51a","Type":"ContainerStarted","Data":"669713e288a0ce610563f75230e6fd23addb8c042ba73ae44cac3da22bb6677a"} Nov 03 20:20:29.733036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:29.732949 4457 generic.go:356] "Generic (PLEG): container finished" podID="c6ab47a3-5965-48cb-b127-4c927ceaf51a" containerID="1ad13c4e09308f4c437f09b11f20ed148b57f7e19c3c7491de04ebbfc8b326e0" exitCode=0 Nov 03 20:20:29.733476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:29.733025 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" event={"ID":"c6ab47a3-5965-48cb-b127-4c927ceaf51a","Type":"ContainerDied","Data":"1ad13c4e09308f4c437f09b11f20ed148b57f7e19c3c7491de04ebbfc8b326e0"} Nov 03 20:20:30.930244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:30.930215 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" Nov 03 20:20:31.034773 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:31.034694 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ab47a3-5965-48cb-b127-4c927ceaf51a-kube-api-access-qcm46\" (UniqueName: \"kubernetes.io/projected/c6ab47a3-5965-48cb-b127-4c927ceaf51a-kube-api-access-qcm46\") pod \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\" (UID: \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\") " Nov 03 20:20:31.034773 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:31.034762 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-openshift-ca\") pod \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\" (UID: \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\") " Nov 03 20:20:31.034966 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:31.034945 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-xks-ca\") pod \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\" (UID: \"c6ab47a3-5965-48cb-b127-4c927ceaf51a\") " Nov 03 20:20:31.035181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:31.035155 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-openshift-ca" pod "c6ab47a3-5965-48cb-b127-4c927ceaf51a" (UID: "c6ab47a3-5965-48cb-b127-4c927ceaf51a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:20:31.035701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:31.035676 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:31.036890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:31.036863 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-xks-ca" pod "c6ab47a3-5965-48cb-b127-4c927ceaf51a" (UID: "c6ab47a3-5965-48cb-b127-4c927ceaf51a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:20:31.037110 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:31.037081 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ab47a3-5965-48cb-b127-4c927ceaf51a-kube-api-access-qcm46" pod "c6ab47a3-5965-48cb-b127-4c927ceaf51a" (UID: "c6ab47a3-5965-48cb-b127-4c927ceaf51a"). InnerVolumeSpecName "kube-api-access-qcm46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:20:31.137311 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:31.137274 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ab47a3-5965-48cb-b127-4c927ceaf51a-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:31.137311 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:31.137306 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcm46\" (UniqueName: \"kubernetes.io/projected/c6ab47a3-5965-48cb-b127-4c927ceaf51a-kube-api-access-qcm46\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:20:31.758610 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:31.758580 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" Nov 03 20:20:31.763408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:31.761583 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf" event={"ID":"c6ab47a3-5965-48cb-b127-4c927ceaf51a","Type":"ContainerDied","Data":"669713e288a0ce610563f75230e6fd23addb8c042ba73ae44cac3da22bb6677a"} Nov 03 20:20:31.763408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:31.761641 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="669713e288a0ce610563f75230e6fd23addb8c042ba73ae44cac3da22bb6677a" Nov 03 20:20:34.029467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:34.029370 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k"] Nov 03 20:20:34.034954 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:34.034925 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cccd04e9-n7z2k"] Nov 03 20:20:35.783333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:35.783295 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91151417-29c0-40d5-80c2-bc8bb5bc5a1b" path="/var/lib/kubelet/pods/91151417-29c0-40d5-80c2-bc8bb5bc5a1b/volumes" Nov 03 20:20:36.187148 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:20:36.187092 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:20:43.029593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:43.029549 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv"] Nov 03 20:20:43.036014 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:43.035978 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58571cf8-lxztv"] Nov 03 20:20:43.764327 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:43.764289 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d059b46-7a38-40ab-9cca-cd51196e9c7e" path="/var/lib/kubelet/pods/0d059b46-7a38-40ab-9cca-cd51196e9c7e/volumes" Nov 03 20:20:45.328217 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.328171 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw"] Nov 03 20:20:45.329440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.329412 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c6ab47a3-5965-48cb-b127-4c927ceaf51a" containerName="revoke-keys" Nov 03 20:20:45.329440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.329438 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ab47a3-5965-48cb-b127-4c927ceaf51a" containerName="revoke-keys" Nov 03 20:20:45.329730 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.329711 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ab47a3-5965-48cb-b127-4c927ceaf51a" containerName="revoke-keys" Nov 03 20:20:45.339779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.339722 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" Nov 03 20:20:45.341743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.341717 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw"] Nov 03 20:20:45.348503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.347605 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2d5927-dockercfg-cj2qx\"" Nov 03 20:20:45.358644 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.358622 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2d5927\"" Nov 03 20:20:45.436417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.419568 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa9ebd7-1105-40cc-b835-344cf70328e0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw\" (UID: \"baa9ebd7-1105-40cc-b835-344cf70328e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" Nov 03 20:20:45.437209 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.437163 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t74qr\" (UniqueName: \"kubernetes.io/projected/baa9ebd7-1105-40cc-b835-344cf70328e0-kube-api-access-t74qr\") pod \"payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw\" (UID: \"baa9ebd7-1105-40cc-b835-344cf70328e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" Nov 03 20:20:45.572409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.570520 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t74qr\" (UniqueName: \"kubernetes.io/projected/baa9ebd7-1105-40cc-b835-344cf70328e0-kube-api-access-t74qr\") pod \"payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw\" (UID: \"baa9ebd7-1105-40cc-b835-344cf70328e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" Nov 03 20:20:45.572409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.570647 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa9ebd7-1105-40cc-b835-344cf70328e0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw\" (UID: \"baa9ebd7-1105-40cc-b835-344cf70328e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" Nov 03 20:20:45.652473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.625020 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa9ebd7-1105-40cc-b835-344cf70328e0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw\" (UID: \"baa9ebd7-1105-40cc-b835-344cf70328e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" Nov 03 20:20:45.652745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.649490 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t74qr\" (UniqueName: \"kubernetes.io/projected/baa9ebd7-1105-40cc-b835-344cf70328e0-kube-api-access-t74qr\") pod \"payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw\" (UID: \"baa9ebd7-1105-40cc-b835-344cf70328e0\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" Nov 03 20:20:45.679013 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:45.672848 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" Nov 03 20:20:46.065438 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:20:46.063492 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa9ebd7_1105_40cc_b835_344cf70328e0.slice/crio-57fa1877ada671f46d742035a56f8f26bcdba7939804de6f8e54f364b4e3dd12 WatchSource:0}: Error finding container 57fa1877ada671f46d742035a56f8f26bcdba7939804de6f8e54f364b4e3dd12: Status 404 returned error can't find the container with id 57fa1877ada671f46d742035a56f8f26bcdba7939804de6f8e54f364b4e3dd12 Nov 03 20:20:46.070905 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:46.070871 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw"] Nov 03 20:20:47.014972 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:47.014927 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" event={"ID":"baa9ebd7-1105-40cc-b835-344cf70328e0","Type":"ContainerStarted","Data":"ee764969818185f2f08a3dd20b47da573dd2a2999b1cb0ea993c157386a1aa8e"} Nov 03 20:20:47.015467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:47.015016 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" event={"ID":"baa9ebd7-1105-40cc-b835-344cf70328e0","Type":"ContainerStarted","Data":"57fa1877ada671f46d742035a56f8f26bcdba7939804de6f8e54f364b4e3dd12"} Nov 03 20:20:47.015467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:47.015061 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" Nov 03 20:20:47.036356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:47.036298 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" podStartSLOduration=2.036283037 podStartE2EDuration="2.036283037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:20:47.034752397 +0000 UTC m=+1425.410349144" watchObservedRunningTime="2026-11-03 20:20:47.036283037 +0000 UTC m=+1425.411879785" Nov 03 20:20:48.054946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:48.052445 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz"] Nov 03 20:20:48.074280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:48.074236 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad10b3a-gzhqz"] Nov 03 20:20:48.653959 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:48.653916 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d"] Nov 03 20:20:48.661972 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:48.661929 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-7ea43c-9xc2d"] Nov 03 20:20:49.764594 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:49.764552 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06817c43-6630-44cb-ad56-f5c4a8e04f25" path="/var/lib/kubelet/pods/06817c43-6630-44cb-ad56-f5c4a8e04f25/volumes" Nov 03 20:20:49.765300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:49.765281 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb24461a-5b4c-4c13-b817-c2c7f3a02b94" path="/var/lib/kubelet/pods/eb24461a-5b4c-4c13-b817-c2c7f3a02b94/volumes" Nov 03 20:20:56.364419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:56.364366 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l"] Nov 03 20:20:56.365260 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:56.365225 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" containerID="cri-o://5ca8587907e5d38822cc1e455d1dc14cf611964d6deebcd6c87a8252100b28e2" gracePeriod=30 Nov 03 20:20:56.918577 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:56.917487 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4"] Nov 03 20:20:56.928603 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:56.928563 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4"] Nov 03 20:20:56.928759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:56.928663 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" Nov 03 20:20:56.932543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:56.932516 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:20:56.932796 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:56.932777 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:20:57.003362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:57.003311 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4\" (UID: \"c989bd63-07c7-4859-91ed-93d59fadebb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" Nov 03 20:20:57.003667 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:57.003650 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwdw2\" (UniqueName: \"kubernetes.io/projected/c989bd63-07c7-4859-91ed-93d59fadebb4-kube-api-access-cwdw2\") pod \"maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4\" (UID: \"c989bd63-07c7-4859-91ed-93d59fadebb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" Nov 03 20:20:57.003800 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:57.003756 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4\" (UID: \"c989bd63-07c7-4859-91ed-93d59fadebb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" Nov 03 20:20:57.104829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:57.104788 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4\" (UID: \"c989bd63-07c7-4859-91ed-93d59fadebb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" Nov 03 20:20:57.105000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:57.104934 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4\" (UID: \"c989bd63-07c7-4859-91ed-93d59fadebb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" Nov 03 20:20:57.105175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:57.105156 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwdw2\" (UniqueName: \"kubernetes.io/projected/c989bd63-07c7-4859-91ed-93d59fadebb4-kube-api-access-cwdw2\") pod \"maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4\" (UID: \"c989bd63-07c7-4859-91ed-93d59fadebb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" Nov 03 20:20:57.105729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:57.105705 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4\" (UID: \"c989bd63-07c7-4859-91ed-93d59fadebb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" Nov 03 20:20:57.107640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:57.107615 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4\" (UID: \"c989bd63-07c7-4859-91ed-93d59fadebb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" Nov 03 20:20:57.120803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:57.120771 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwdw2\" (UniqueName: \"kubernetes.io/projected/c989bd63-07c7-4859-91ed-93d59fadebb4-kube-api-access-cwdw2\") pod \"maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4\" (UID: \"c989bd63-07c7-4859-91ed-93d59fadebb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" Nov 03 20:20:57.255465 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:57.251695 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" Nov 03 20:20:57.720877 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:20:57.720848 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc989bd63_07c7_4859_91ed_93d59fadebb4.slice/crio-c634a7cf69dd61fa09932a742c9b0a8f99f1bc5c09916e51f7dc7fed9194d56a WatchSource:0}: Error finding container c634a7cf69dd61fa09932a742c9b0a8f99f1bc5c09916e51f7dc7fed9194d56a: Status 404 returned error can't find the container with id c634a7cf69dd61fa09932a742c9b0a8f99f1bc5c09916e51f7dc7fed9194d56a Nov 03 20:20:57.722674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:57.722652 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4"] Nov 03 20:20:58.072702 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:58.072678 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" Nov 03 20:20:58.177325 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:58.177290 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" event={"ID":"c989bd63-07c7-4859-91ed-93d59fadebb4","Type":"ContainerStarted","Data":"c634a7cf69dd61fa09932a742c9b0a8f99f1bc5c09916e51f7dc7fed9194d56a"} Nov 03 20:20:59.319362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:59.318851 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl"] Nov 03 20:20:59.685548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:59.683405 4457 generic.go:356] "Generic (PLEG): container finished" podID="c989bd63-07c7-4859-91ed-93d59fadebb4" containerID="72025990159b59d0d715c66f59d6a42348872aef070cdea19ca0adff51c96656" exitCode=0 Nov 03 20:20:59.685548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:59.683798 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" event={"ID":"c989bd63-07c7-4859-91ed-93d59fadebb4","Type":"ContainerDied","Data":"72025990159b59d0d715c66f59d6a42348872aef070cdea19ca0adff51c96656"} Nov 03 20:20:59.762411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:20:59.753728 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9886285f-sx9vl"] Nov 03 20:21:00.054897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:00.052918 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a0479f-d3fd-4919-9bac-d69caaa62449" path="/var/lib/kubelet/pods/33a0479f-d3fd-4919-9bac-d69caaa62449/volumes" Nov 03 20:21:00.202430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:00.188261 4457 patch_prober.go:29] interesting pod/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:21:00.202430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:00.188327 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:21:00.911541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:00.911490 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.023556 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.022496 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.039578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.039499 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.053586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.053517 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:21:01.056082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.055619 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:21:01.142412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.139568 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.429411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.389513 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 03 20:21:01.429411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.415473 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.429411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.419153 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.430544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.430504 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.449132 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.449093 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:21:01.573616 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.573570 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 03 20:21:01.684581 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.618254 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:01.794863 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.784974 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:21:01.913414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:01.912577 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:02.054471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.044177 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:02.083314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.072449 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:02.166074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.166004 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:02.353206 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.353125 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:21:02.353484 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.353463 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:21:02.353650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.353635 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:21:02.353802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.353787 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:21:02.353948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.353933 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:21:02.357804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.355042 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" Nov 03 20:21:02.406413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.380047 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" event={"ID":"c989bd63-07c7-4859-91ed-93d59fadebb4","Type":"ContainerDied","Data":"c634a7cf69dd61fa09932a742c9b0a8f99f1bc5c09916e51f7dc7fed9194d56a"} Nov 03 20:21:02.406413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.380098 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c634a7cf69dd61fa09932a742c9b0a8f99f1bc5c09916e51f7dc7fed9194d56a" Nov 03 20:21:02.406413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.387668 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:21:02.447411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.419738 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:21:02.447411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.422220 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:21:02.459411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.450851 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:21:02.459411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.452273 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:21:02.499419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.484141 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-xks-ca\") pod \"c989bd63-07c7-4859-91ed-93d59fadebb4\" (UID: \"c989bd63-07c7-4859-91ed-93d59fadebb4\") " Nov 03 20:21:02.499419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.484233 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-openshift-ca\") pod \"c989bd63-07c7-4859-91ed-93d59fadebb4\" (UID: \"c989bd63-07c7-4859-91ed-93d59fadebb4\") " Nov 03 20:21:02.499419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.485266 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c989bd63-07c7-4859-91ed-93d59fadebb4-kube-api-access-cwdw2\" (UniqueName: \"kubernetes.io/projected/c989bd63-07c7-4859-91ed-93d59fadebb4-kube-api-access-cwdw2\") pod \"c989bd63-07c7-4859-91ed-93d59fadebb4\" (UID: \"c989bd63-07c7-4859-91ed-93d59fadebb4\") " Nov 03 20:21:02.499419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.491624 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-openshift-ca" pod "c989bd63-07c7-4859-91ed-93d59fadebb4" (UID: "c989bd63-07c7-4859-91ed-93d59fadebb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:02.612613 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.506053 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c989bd63-07c7-4859-91ed-93d59fadebb4-kube-api-access-cwdw2" pod "c989bd63-07c7-4859-91ed-93d59fadebb4" (UID: "c989bd63-07c7-4859-91ed-93d59fadebb4"). InnerVolumeSpecName "kube-api-access-cwdw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:02.689621 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.685639 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-xks-ca" pod "c989bd63-07c7-4859-91ed-93d59fadebb4" (UID: "c989bd63-07c7-4859-91ed-93d59fadebb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:21:02.754414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.752598 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwdw2\" (UniqueName: \"kubernetes.io/projected/c989bd63-07c7-4859-91ed-93d59fadebb4-kube-api-access-cwdw2\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:02.754414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.752642 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:02.754414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.752664 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c989bd63-07c7-4859-91ed-93d59fadebb4-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:02.862413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.859090 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:21:02.945463 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:02.933462 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:21:03.080413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.078014 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:21:03.146063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.146027 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:21:03.447912 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.441624 4457 generic.go:356] "Generic (PLEG): container finished" podID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerID="5ca8587907e5d38822cc1e455d1dc14cf611964d6deebcd6c87a8252100b28e2" exitCode=0 Nov 03 20:21:03.447912 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.442709 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" event={"ID":"90af0b2d-03b0-4532-85ee-f3282cc9637a","Type":"ContainerDied","Data":"5ca8587907e5d38822cc1e455d1dc14cf611964d6deebcd6c87a8252100b28e2"} Nov 03 20:21:03.447912 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.443121 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4" Nov 03 20:21:03.447912 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.444474 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:21:03.451662 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.451635 4457 scope.go:122] "RemoveContainer" containerID="9997f4ec20b8ebad6e4d59ad71e3d378339d5ca88f1b29495739890a9a50a094" Nov 03 20:21:03.553243 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.553197 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-podinfo\") pod \"90af0b2d-03b0-4532-85ee-f3282cc9637a\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " Nov 03 20:21:03.553603 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.553581 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-socket\") pod \"90af0b2d-03b0-4532-85ee-f3282cc9637a\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " Nov 03 20:21:03.553807 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.553790 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-kube-api-access-z2qvf\" (UniqueName: \"kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-kube-api-access-z2qvf\") pod \"90af0b2d-03b0-4532-85ee-f3282cc9637a\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " Nov 03 20:21:03.556625 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.556592 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-certs\") pod \"90af0b2d-03b0-4532-85ee-f3282cc9637a\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " Nov 03 20:21:03.556774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.556701 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-data\") pod \"90af0b2d-03b0-4532-85ee-f3282cc9637a\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " Nov 03 20:21:03.556918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.556875 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90af0b2d-03b0-4532-85ee-f3282cc9637a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90af0b2d-03b0-4532-85ee-f3282cc9637a-istiod-ca-cert\") pod \"90af0b2d-03b0-4532-85ee-f3282cc9637a\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " Nov 03 20:21:03.557046 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.556953 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-envoy\") pod \"90af0b2d-03b0-4532-85ee-f3282cc9637a\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " Nov 03 20:21:03.557134 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.557089 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-credential-socket\") pod \"90af0b2d-03b0-4532-85ee-f3282cc9637a\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " Nov 03 20:21:03.557239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.557201 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-token\" (UniqueName: \"kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-token\") pod \"90af0b2d-03b0-4532-85ee-f3282cc9637a\" (UID: \"90af0b2d-03b0-4532-85ee-f3282cc9637a\") " Nov 03 20:21:03.560970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.560685 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-socket" pod "90af0b2d-03b0-4532-85ee-f3282cc9637a" (UID: "90af0b2d-03b0-4532-85ee-f3282cc9637a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:03.561084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.560702 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-data" pod "90af0b2d-03b0-4532-85ee-f3282cc9637a" (UID: "90af0b2d-03b0-4532-85ee-f3282cc9637a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:03.561084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.560903 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-credential-socket" pod "90af0b2d-03b0-4532-85ee-f3282cc9637a" (UID: "90af0b2d-03b0-4532-85ee-f3282cc9637a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:03.561245 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.561222 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90af0b2d-03b0-4532-85ee-f3282cc9637a-istiod-ca-cert" pod "90af0b2d-03b0-4532-85ee-f3282cc9637a" (UID: "90af0b2d-03b0-4532-85ee-f3282cc9637a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:03.562650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.562625 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-certs" pod "90af0b2d-03b0-4532-85ee-f3282cc9637a" (UID: "90af0b2d-03b0-4532-85ee-f3282cc9637a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:03.563137 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.563111 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-token" pod "90af0b2d-03b0-4532-85ee-f3282cc9637a" (UID: "90af0b2d-03b0-4532-85ee-f3282cc9637a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:03.563293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.563272 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-envoy" pod "90af0b2d-03b0-4532-85ee-f3282cc9637a" (UID: "90af0b2d-03b0-4532-85ee-f3282cc9637a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:03.574433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.571900 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-kube-api-access-z2qvf" pod "90af0b2d-03b0-4532-85ee-f3282cc9637a" (UID: "90af0b2d-03b0-4532-85ee-f3282cc9637a"). InnerVolumeSpecName "kube-api-access-z2qvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:03.574433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.572488 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-podinfo" pod "90af0b2d-03b0-4532-85ee-f3282cc9637a" (UID: "90af0b2d-03b0-4532-85ee-f3282cc9637a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:03.694438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.694280 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:03.694438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.694328 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:03.694438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.694343 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90af0b2d-03b0-4532-85ee-f3282cc9637a-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:03.694438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.694358 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:03.694438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.694372 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:03.694438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.694404 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:03.694438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.694421 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90af0b2d-03b0-4532-85ee-f3282cc9637a-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:03.694438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.694442 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90af0b2d-03b0-4532-85ee-f3282cc9637a-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:03.695059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:03.694456 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2qvf\" (UniqueName: \"kubernetes.io/projected/90af0b2d-03b0-4532-85ee-f3282cc9637a-kube-api-access-z2qvf\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:04.508420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:04.505534 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" event={"ID":"90af0b2d-03b0-4532-85ee-f3282cc9637a","Type":"ContainerDied","Data":"2710ed5ab459be5ae49b231e2a3e0ea3fe11616c3a04c9e90c64dec6de168cbd"} Nov 03 20:21:04.508420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:04.505643 4457 scope.go:122] "RemoveContainer" containerID="5ca8587907e5d38822cc1e455d1dc14cf611964d6deebcd6c87a8252100b28e2" Nov 03 20:21:04.508420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:04.505823 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l" Nov 03 20:21:04.758079 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:04.755667 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l"] Nov 03 20:21:04.785656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:04.785545 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7ea43c-openshift-default-5867895fd7-4lp9l"] Nov 03 20:21:05.619730 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.619690 4457 scope.go:122] "RemoveContainer" containerID="c9f3ac61b0aa4cbcb9fd29e0ce0c68db6c1815da90f98b2c65c42d6832e4554e" Nov 03 20:21:05.630031 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.630002 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7"] Nov 03 20:21:05.631797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.631664 4457 cpu_manager.go:414] "containerMap: removing container" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" Nov 03 20:21:05.631797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.631692 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" Nov 03 20:21:05.631797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.631716 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c989bd63-07c7-4859-91ed-93d59fadebb4" containerName="revoke-keys" Nov 03 20:21:05.631797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.631724 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c989bd63-07c7-4859-91ed-93d59fadebb4" containerName="revoke-keys" Nov 03 20:21:05.632310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.632291 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" Nov 03 20:21:05.632517 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.632331 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c989bd63-07c7-4859-91ed-93d59fadebb4" containerName="revoke-keys" Nov 03 20:21:05.635774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.635748 4457 scope.go:122] "RemoveContainer" containerID="fac73a045bed2b38fae07233bbe55b24319cda3bc82505c28e02b93adb252dfe" Nov 03 20:21:05.642628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.639568 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" Nov 03 20:21:05.646039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.645370 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-7fd1a7\"" Nov 03 20:21:05.646039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.645575 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-7fd1a7-dockercfg-ml5rb\"" Nov 03 20:21:05.658007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.657983 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7"] Nov 03 20:21:05.681918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.681752 4457 scope.go:122] "RemoveContainer" containerID="33cdcb68dd54521a2f7a158b2205104306cacb53fe276a6bc8568673180aadee" Nov 03 20:21:05.703373 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.702943 4457 scope.go:122] "RemoveContainer" containerID="64ba715241566e4f627d58f10820607d2b8029257345d85dff95f2f6fe969df1" Nov 03 20:21:05.732474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.732432 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j76bm\" (UniqueName: \"kubernetes.io/projected/cc705cd8-f6b5-4907-a90c-39f0c779f227-kube-api-access-j76bm\") pod \"payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7\" (UID: \"cc705cd8-f6b5-4907-a90c-39f0c779f227\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" Nov 03 20:21:05.732732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.732710 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc705cd8-f6b5-4907-a90c-39f0c779f227-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7\" (UID: \"cc705cd8-f6b5-4907-a90c-39f0c779f227\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" Nov 03 20:21:05.843428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.840531 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j76bm\" (UniqueName: \"kubernetes.io/projected/cc705cd8-f6b5-4907-a90c-39f0c779f227-kube-api-access-j76bm\") pod \"payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7\" (UID: \"cc705cd8-f6b5-4907-a90c-39f0c779f227\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" Nov 03 20:21:05.843428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.840957 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc705cd8-f6b5-4907-a90c-39f0c779f227-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7\" (UID: \"cc705cd8-f6b5-4907-a90c-39f0c779f227\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" Nov 03 20:21:05.844403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.844274 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc705cd8-f6b5-4907-a90c-39f0c779f227-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7\" (UID: \"cc705cd8-f6b5-4907-a90c-39f0c779f227\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" Nov 03 20:21:05.938679 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.938247 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j76bm\" (UniqueName: \"kubernetes.io/projected/cc705cd8-f6b5-4907-a90c-39f0c779f227-kube-api-access-j76bm\") pod \"payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7\" (UID: \"cc705cd8-f6b5-4907-a90c-39f0c779f227\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" Nov 03 20:21:05.968706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:05.968672 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" Nov 03 20:21:06.013464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:06.009640 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" path="/var/lib/kubelet/pods/90af0b2d-03b0-4532-85ee-f3282cc9637a/volumes" Nov 03 20:21:06.195214 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:21:06.194246 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:21:06.381956 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:06.381926 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7"] Nov 03 20:21:06.382100 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:21:06.381924 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc705cd8_f6b5_4907_a90c_39f0c779f227.slice/crio-a444961af2e8128cffecddd4471f108818d7802a35a7d2d72aab8e9bb79a6881 WatchSource:0}: Error finding container a444961af2e8128cffecddd4471f108818d7802a35a7d2d72aab8e9bb79a6881: Status 404 returned error can't find the container with id a444961af2e8128cffecddd4471f108818d7802a35a7d2d72aab8e9bb79a6881 Nov 03 20:21:06.531345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:06.531308 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" event={"ID":"cc705cd8-f6b5-4907-a90c-39f0c779f227","Type":"ContainerStarted","Data":"938c78e5fd248904c6b7d217fee2fa51bb64397dd0ddf1f51bcd37ccd6286871"} Nov 03 20:21:06.531538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:06.531379 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" Nov 03 20:21:06.531538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:06.531443 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" event={"ID":"cc705cd8-f6b5-4907-a90c-39f0c779f227","Type":"ContainerStarted","Data":"a444961af2e8128cffecddd4471f108818d7802a35a7d2d72aab8e9bb79a6881"} Nov 03 20:21:07.020310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:07.020248 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" podStartSLOduration=2.020228538 podStartE2EDuration="2.020228538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:06.550197457 +0000 UTC m=+1444.925794203" watchObservedRunningTime="2026-11-03 20:21:07.020228538 +0000 UTC m=+1445.395825285" Nov 03 20:21:07.027631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:07.027589 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz"] Nov 03 20:21:08.029116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:08.029081 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz"] Nov 03 20:21:08.035367 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:08.035339 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-012c1ed0-d2mkz"] Nov 03 20:21:09.795793 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:09.795297 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70996ecc-78bc-49e7-be91-83904b62094a" path="/var/lib/kubelet/pods/70996ecc-78bc-49e7-be91-83904b62094a/volumes" Nov 03 20:21:11.043166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.043115 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb"] Nov 03 20:21:11.045102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.045071 4457 cpu_manager.go:414] "containerMap: removing container" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" Nov 03 20:21:11.045102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.045101 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" Nov 03 20:21:11.045335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.045290 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="90af0b2d-03b0-4532-85ee-f3282cc9637a" containerName="istio-proxy" Nov 03 20:21:11.052221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.052186 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" Nov 03 20:21:11.055340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.055308 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb"] Nov 03 20:21:11.068701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.065665 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:21:11.068701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.065713 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:21:11.233922 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.233883 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb\" (UID: \"6f0384e1-5813-48bc-9cd4-184ce22ca486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" Nov 03 20:21:11.234099 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.233995 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb\" (UID: \"6f0384e1-5813-48bc-9cd4-184ce22ca486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" Nov 03 20:21:11.234263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.234237 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l779d\" (UniqueName: \"kubernetes.io/projected/6f0384e1-5813-48bc-9cd4-184ce22ca486-kube-api-access-l779d\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb\" (UID: \"6f0384e1-5813-48bc-9cd4-184ce22ca486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" Nov 03 20:21:11.336105 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.336072 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l779d\" (UniqueName: \"kubernetes.io/projected/6f0384e1-5813-48bc-9cd4-184ce22ca486-kube-api-access-l779d\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb\" (UID: \"6f0384e1-5813-48bc-9cd4-184ce22ca486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" Nov 03 20:21:11.336289 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.336170 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb\" (UID: \"6f0384e1-5813-48bc-9cd4-184ce22ca486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" Nov 03 20:21:11.336289 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.336251 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb\" (UID: \"6f0384e1-5813-48bc-9cd4-184ce22ca486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" Nov 03 20:21:11.336873 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.336852 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb\" (UID: \"6f0384e1-5813-48bc-9cd4-184ce22ca486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" Nov 03 20:21:11.338837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.338808 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb\" (UID: \"6f0384e1-5813-48bc-9cd4-184ce22ca486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" Nov 03 20:21:11.348567 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.348543 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l779d\" (UniqueName: \"kubernetes.io/projected/6f0384e1-5813-48bc-9cd4-184ce22ca486-kube-api-access-l779d\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb\" (UID: \"6f0384e1-5813-48bc-9cd4-184ce22ca486\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" Nov 03 20:21:11.493099 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.492818 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" Nov 03 20:21:11.817089 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:21:11.817055 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0384e1_5813_48bc_9cd4_184ce22ca486.slice/crio-3ff5231c3748b684da8500a254ec087332cc99acb2fc24cdddf9252a0fe690b1 WatchSource:0}: Error finding container 3ff5231c3748b684da8500a254ec087332cc99acb2fc24cdddf9252a0fe690b1: Status 404 returned error can't find the container with id 3ff5231c3748b684da8500a254ec087332cc99acb2fc24cdddf9252a0fe690b1 Nov 03 20:21:11.820305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:11.820280 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb"] Nov 03 20:21:12.029058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.028943 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" podUID="d1acef2b-8495-41ff-8a07-1797e7a8a1a6" containerName="payload-pre-processing" containerID="cri-o://d9ccab01694976ba56bb7474aa9c6ce4acd8b3daeac836ad15f66e1a2e6f9f5f" gracePeriod=25 Nov 03 20:21:12.514690 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.514586 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" Nov 03 20:21:12.525439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.524784 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-plugins-config-volume\") pod \"d1acef2b-8495-41ff-8a07-1797e7a8a1a6\" (UID: \"d1acef2b-8495-41ff-8a07-1797e7a8a1a6\") " Nov 03 20:21:12.525439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.525028 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-kube-api-access-r7jsq\" (UniqueName: \"kubernetes.io/projected/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-kube-api-access-r7jsq\") pod \"d1acef2b-8495-41ff-8a07-1797e7a8a1a6\" (UID: \"d1acef2b-8495-41ff-8a07-1797e7a8a1a6\") " Nov 03 20:21:12.525439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.525168 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-plugins-config-volume" pod "d1acef2b-8495-41ff-8a07-1797e7a8a1a6" (UID: "d1acef2b-8495-41ff-8a07-1797e7a8a1a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:12.525752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.525729 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:12.527468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.527435 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-kube-api-access-r7jsq" pod "d1acef2b-8495-41ff-8a07-1797e7a8a1a6" (UID: "d1acef2b-8495-41ff-8a07-1797e7a8a1a6"). InnerVolumeSpecName "kube-api-access-r7jsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:12.627135 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.627053 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7jsq\" (UniqueName: \"kubernetes.io/projected/d1acef2b-8495-41ff-8a07-1797e7a8a1a6-kube-api-access-r7jsq\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:12.634061 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.634022 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" event={"ID":"6f0384e1-5813-48bc-9cd4-184ce22ca486","Type":"ContainerStarted","Data":"57a0b7b15f2d6e16a194da47dd6698f8ddc70f00db16a53c884b8cfe852ce027"} Nov 03 20:21:12.634250 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.634122 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" event={"ID":"6f0384e1-5813-48bc-9cd4-184ce22ca486","Type":"ContainerStarted","Data":"3ff5231c3748b684da8500a254ec087332cc99acb2fc24cdddf9252a0fe690b1"} Nov 03 20:21:12.635758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.635730 4457 generic.go:356] "Generic (PLEG): container finished" podID="d1acef2b-8495-41ff-8a07-1797e7a8a1a6" containerID="d9ccab01694976ba56bb7474aa9c6ce4acd8b3daeac836ad15f66e1a2e6f9f5f" exitCode=0 Nov 03 20:21:12.635900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.635765 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" event={"ID":"d1acef2b-8495-41ff-8a07-1797e7a8a1a6","Type":"ContainerDied","Data":"d9ccab01694976ba56bb7474aa9c6ce4acd8b3daeac836ad15f66e1a2e6f9f5f"} Nov 03 20:21:12.635900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.635773 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" Nov 03 20:21:12.635900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.635796 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz" event={"ID":"d1acef2b-8495-41ff-8a07-1797e7a8a1a6","Type":"ContainerDied","Data":"8d49ed7796074448ca8c7be157e8808b215b01a3c3fb8349443ffac86374233e"} Nov 03 20:21:12.635900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.635817 4457 scope.go:122] "RemoveContainer" containerID="d9ccab01694976ba56bb7474aa9c6ce4acd8b3daeac836ad15f66e1a2e6f9f5f" Nov 03 20:21:12.648283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.648253 4457 scope.go:122] "RemoveContainer" containerID="d9ccab01694976ba56bb7474aa9c6ce4acd8b3daeac836ad15f66e1a2e6f9f5f" Nov 03 20:21:12.648586 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:21:12.648563 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ccab01694976ba56bb7474aa9c6ce4acd8b3daeac836ad15f66e1a2e6f9f5f\": container with ID starting with d9ccab01694976ba56bb7474aa9c6ce4acd8b3daeac836ad15f66e1a2e6f9f5f not found: ID does not exist" containerID="d9ccab01694976ba56bb7474aa9c6ce4acd8b3daeac836ad15f66e1a2e6f9f5f" Nov 03 20:21:12.648670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.648596 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ccab01694976ba56bb7474aa9c6ce4acd8b3daeac836ad15f66e1a2e6f9f5f"} err="failed to get container status \"d9ccab01694976ba56bb7474aa9c6ce4acd8b3daeac836ad15f66e1a2e6f9f5f\": rpc error: code = NotFound desc = could not find container \"d9ccab01694976ba56bb7474aa9c6ce4acd8b3daeac836ad15f66e1a2e6f9f5f\": container with ID starting with d9ccab01694976ba56bb7474aa9c6ce4acd8b3daeac836ad15f66e1a2e6f9f5f not found: ID does not exist" Nov 03 20:21:12.678028 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.656232 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" podStartSLOduration=1.656215538 podStartE2EDuration="1.656215538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:12.653578153 +0000 UTC m=+1451.029174901" watchObservedRunningTime="2026-11-03 20:21:12.656215538 +0000 UTC m=+1451.031812287" Nov 03 20:21:12.696457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.695862 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz"] Nov 03 20:21:12.703917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:12.703877 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9af0cd-7968fc9587-k2pxz"] Nov 03 20:21:13.025996 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:13.025898 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq"] Nov 03 20:21:13.030841 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:13.030808 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-301728c8-7vksq"] Nov 03 20:21:13.647745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:13.647708 4457 generic.go:356] "Generic (PLEG): container finished" podID="6f0384e1-5813-48bc-9cd4-184ce22ca486" containerID="57a0b7b15f2d6e16a194da47dd6698f8ddc70f00db16a53c884b8cfe852ce027" exitCode=0 Nov 03 20:21:13.648257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:13.647797 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" event={"ID":"6f0384e1-5813-48bc-9cd4-184ce22ca486","Type":"ContainerDied","Data":"57a0b7b15f2d6e16a194da47dd6698f8ddc70f00db16a53c884b8cfe852ce027"} Nov 03 20:21:13.761572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:13.761535 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e6551b5-3042-42e0-a09c-e8ec9b50be1f" path="/var/lib/kubelet/pods/0e6551b5-3042-42e0-a09c-e8ec9b50be1f/volumes" Nov 03 20:21:13.762209 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:13.762191 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1acef2b-8495-41ff-8a07-1797e7a8a1a6" path="/var/lib/kubelet/pods/d1acef2b-8495-41ff-8a07-1797e7a8a1a6/volumes" Nov 03 20:21:14.922890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:14.922810 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" Nov 03 20:21:14.979581 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:14.979539 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-openshift-ca\") pod \"6f0384e1-5813-48bc-9cd4-184ce22ca486\" (UID: \"6f0384e1-5813-48bc-9cd4-184ce22ca486\") " Nov 03 20:21:14.979770 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:14.979639 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0384e1-5813-48bc-9cd4-184ce22ca486-kube-api-access-l779d\" (UniqueName: \"kubernetes.io/projected/6f0384e1-5813-48bc-9cd4-184ce22ca486-kube-api-access-l779d\") pod \"6f0384e1-5813-48bc-9cd4-184ce22ca486\" (UID: \"6f0384e1-5813-48bc-9cd4-184ce22ca486\") " Nov 03 20:21:14.979876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:14.979843 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-xks-ca\") pod \"6f0384e1-5813-48bc-9cd4-184ce22ca486\" (UID: \"6f0384e1-5813-48bc-9cd4-184ce22ca486\") " Nov 03 20:21:14.980003 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:14.979970 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-openshift-ca" pod "6f0384e1-5813-48bc-9cd4-184ce22ca486" (UID: "6f0384e1-5813-48bc-9cd4-184ce22ca486"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:14.980591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:14.980563 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:14.985440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:14.982747 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-xks-ca" pod "6f0384e1-5813-48bc-9cd4-184ce22ca486" (UID: "6f0384e1-5813-48bc-9cd4-184ce22ca486"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:21:14.985440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:14.983580 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0384e1-5813-48bc-9cd4-184ce22ca486-kube-api-access-l779d" pod "6f0384e1-5813-48bc-9cd4-184ce22ca486" (UID: "6f0384e1-5813-48bc-9cd4-184ce22ca486"). InnerVolumeSpecName "kube-api-access-l779d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:15.083145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:15.083104 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f0384e1-5813-48bc-9cd4-184ce22ca486-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:15.083145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:15.083148 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l779d\" (UniqueName: \"kubernetes.io/projected/6f0384e1-5813-48bc-9cd4-184ce22ca486-kube-api-access-l779d\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:15.678293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:15.675658 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" Nov 03 20:21:15.678293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:15.675743 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb" event={"ID":"6f0384e1-5813-48bc-9cd4-184ce22ca486","Type":"ContainerDied","Data":"3ff5231c3748b684da8500a254ec087332cc99acb2fc24cdddf9252a0fe690b1"} Nov 03 20:21:15.678293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:15.675818 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ff5231c3748b684da8500a254ec087332cc99acb2fc24cdddf9252a0fe690b1" Nov 03 20:21:17.546042 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:17.546014 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" Nov 03 20:21:17.774153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:17.774125 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg"] Nov 03 20:21:17.774364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:17.774341 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" containerID="cri-o://5aa6f4e731589257eb2a9efe391c3d2891d83ef37a4a05c68d25049f89219b8b" gracePeriod=30 Nov 03 20:21:18.730371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:18.730331 4457 generic.go:356] "Generic (PLEG): container finished" podID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerID="94d8811fd536c2ba120cffa921554030600eeed3a885381a00c87370091c9fe1" exitCode=0 Nov 03 20:21:18.730371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:18.730412 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" event={"ID":"09c79f72-431b-4137-950f-b5f8d68f9a9e","Type":"ContainerDied","Data":"94d8811fd536c2ba120cffa921554030600eeed3a885381a00c87370091c9fe1"} Nov 03 20:21:18.730938 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:18.730881 4457 scope.go:122] "RemoveContainer" containerID="94d8811fd536c2ba120cffa921554030600eeed3a885381a00c87370091c9fe1" Nov 03 20:21:18.732866 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:18.732833 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:18.732975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:18.732909 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:18.732975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:18.732942 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:21:18.798455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:18.798411 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:21:19.833442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:19.832647 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" event={"ID":"09c79f72-431b-4137-950f-b5f8d68f9a9e","Type":"ContainerStarted","Data":"4023e946742b90a818c32e9a3b61577d9adb17fe2f0e5b34431c2d802b87f940"} Nov 03 20:21:19.956408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:19.956293 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:21:19.964330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:19.962701 4457 patch_prober.go:29] interesting pod/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:21:19.964330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:19.962758 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 03 20:21:20.947411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:20.947317 4457 patch_prober.go:29] interesting pod/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:21:20.947411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:20.947378 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 03 20:21:21.614220 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:21.614178 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn"] Nov 03 20:21:21.625284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:21.625249 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t"] Nov 03 20:21:21.946862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:21.946774 4457 patch_prober.go:29] interesting pod/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:21:21.952473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:21.950451 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 03 20:21:22.942285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:22.942242 4457 generic.go:356] "Generic (PLEG): container finished" podID="27abf71d-0887-4f90-a511-54719c9194db" containerID="5aa6f4e731589257eb2a9efe391c3d2891d83ef37a4a05c68d25049f89219b8b" exitCode=0 Nov 03 20:21:22.942498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:22.942359 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" event={"ID":"27abf71d-0887-4f90-a511-54719c9194db","Type":"ContainerDied","Data":"5aa6f4e731589257eb2a9efe391c3d2891d83ef37a4a05c68d25049f89219b8b"} Nov 03 20:21:22.944235 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:22.944208 4457 patch_prober.go:29] interesting pod/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:21:22.944480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:22.944451 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 03 20:21:23.651053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.651022 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:21:23.725684 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.725636 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-envoy\") pod \"27abf71d-0887-4f90-a511-54719c9194db\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " Nov 03 20:21:23.727364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.725923 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-istio-token\" (UniqueName: \"kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-istio-token\") pod \"27abf71d-0887-4f90-a511-54719c9194db\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " Nov 03 20:21:23.727364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.725971 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-credential-socket\") pod \"27abf71d-0887-4f90-a511-54719c9194db\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " Nov 03 20:21:23.727364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.726102 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-certs\") pod \"27abf71d-0887-4f90-a511-54719c9194db\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " Nov 03 20:21:23.727364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.726192 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-kube-api-access-k8l5k\" (UniqueName: \"kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-kube-api-access-k8l5k\") pod \"27abf71d-0887-4f90-a511-54719c9194db\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " Nov 03 20:21:23.727364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.726326 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-socket\") pod \"27abf71d-0887-4f90-a511-54719c9194db\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " Nov 03 20:21:23.727364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.726493 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27abf71d-0887-4f90-a511-54719c9194db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27abf71d-0887-4f90-a511-54719c9194db-istio-podinfo\") pod \"27abf71d-0887-4f90-a511-54719c9194db\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " Nov 03 20:21:23.727364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.726612 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-data\") pod \"27abf71d-0887-4f90-a511-54719c9194db\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " Nov 03 20:21:23.727364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.726719 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27abf71d-0887-4f90-a511-54719c9194db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27abf71d-0887-4f90-a511-54719c9194db-istiod-ca-cert\") pod \"27abf71d-0887-4f90-a511-54719c9194db\" (UID: \"27abf71d-0887-4f90-a511-54719c9194db\") " Nov 03 20:21:23.735768 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.728741 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-credential-socket" pod "27abf71d-0887-4f90-a511-54719c9194db" (UID: "27abf71d-0887-4f90-a511-54719c9194db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.735768 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.728924 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-socket" pod "27abf71d-0887-4f90-a511-54719c9194db" (UID: "27abf71d-0887-4f90-a511-54719c9194db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.735768 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.729047 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-data" pod "27abf71d-0887-4f90-a511-54719c9194db" (UID: "27abf71d-0887-4f90-a511-54719c9194db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.735768 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.729498 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27abf71d-0887-4f90-a511-54719c9194db-istiod-ca-cert" pod "27abf71d-0887-4f90-a511-54719c9194db" (UID: "27abf71d-0887-4f90-a511-54719c9194db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:23.735768 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.729556 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-certs" pod "27abf71d-0887-4f90-a511-54719c9194db" (UID: "27abf71d-0887-4f90-a511-54719c9194db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.737738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.737581 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-kube-api-access-k8l5k" pod "27abf71d-0887-4f90-a511-54719c9194db" (UID: "27abf71d-0887-4f90-a511-54719c9194db"). InnerVolumeSpecName "kube-api-access-k8l5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:23.737738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.737656 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27abf71d-0887-4f90-a511-54719c9194db-istio-podinfo" pod "27abf71d-0887-4f90-a511-54719c9194db" (UID: "27abf71d-0887-4f90-a511-54719c9194db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:23.737738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.737705 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-envoy" pod "27abf71d-0887-4f90-a511-54719c9194db" (UID: "27abf71d-0887-4f90-a511-54719c9194db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:23.738886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.738852 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-istio-token" pod "27abf71d-0887-4f90-a511-54719c9194db" (UID: "27abf71d-0887-4f90-a511-54719c9194db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:23.871526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.854082 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27abf71d-0887-4f90-a511-54719c9194db-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.871526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.854119 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.871526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.854136 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.871526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.854151 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.871526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.854167 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.871526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.854181 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8l5k\" (UniqueName: \"kubernetes.io/projected/27abf71d-0887-4f90-a511-54719c9194db-kube-api-access-k8l5k\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.871526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.854195 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.871526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.854208 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27abf71d-0887-4f90-a511-54719c9194db-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.871526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.854221 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27abf71d-0887-4f90-a511-54719c9194db-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:23.945482 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.945368 4457 patch_prober.go:29] interesting pod/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:21:23.945877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.945508 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 03 20:21:23.961407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.956455 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" event={"ID":"27abf71d-0887-4f90-a511-54719c9194db","Type":"ContainerDied","Data":"74d434c652c3712f5e9e01758b76b89872f3783caf6625123da3d6c9ceab3f34"} Nov 03 20:21:23.961407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.956532 4457 scope.go:122] "RemoveContainer" containerID="5aa6f4e731589257eb2a9efe391c3d2891d83ef37a4a05c68d25049f89219b8b" Nov 03 20:21:23.961407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.956955 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg" Nov 03 20:21:23.997413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:23.994012 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg"] Nov 03 20:21:24.002100 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:24.002075 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-xkcgg"] Nov 03 20:21:24.946408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:24.946118 4457 patch_prober.go:29] interesting pod/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:21:24.946408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:24.946205 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 03 20:21:25.740953 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:25.740911 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4"] Nov 03 20:21:25.748514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:25.748478 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-9af0cd-qvvp4"] Nov 03 20:21:25.786414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:25.785587 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27abf71d-0887-4f90-a511-54719c9194db" path="/var/lib/kubelet/pods/27abf71d-0887-4f90-a511-54719c9194db/volumes" Nov 03 20:21:25.786616 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:25.786555 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c989bd63-07c7-4859-91ed-93d59fadebb4" path="/var/lib/kubelet/pods/c989bd63-07c7-4859-91ed-93d59fadebb4/volumes" Nov 03 20:21:25.945215 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:25.945170 4457 patch_prober.go:29] interesting pod/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 03 20:21:25.945362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:25.945242 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 03 20:21:26.619864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:26.618892 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" podUID="81b5ac0a-b306-458b-8b39-96deb3b575fa" containerName="payload-processing" containerID="cri-o://89904689beceaf5a1e9929e6cccc9d91eeddf5c65db67dfc70958b0600d72440" gracePeriod=25 Nov 03 20:21:26.635769 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:26.634468 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" podUID="c254c14f-13f1-4e20-bd27-14585726663f" containerName="payload-pre-processing" containerID="cri-o://8771c19020b4901bc0c96afa8c44d8dc8e7618713784fb46a9a038d75f81e6ad" gracePeriod=25 Nov 03 20:21:26.952990 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:26.951145 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:21:26.954447 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:26.954273 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:21:26.955262 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:26.955219 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:21:27.132232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.132192 4457 generic.go:356] "Generic (PLEG): container finished" podID="81b5ac0a-b306-458b-8b39-96deb3b575fa" containerID="89904689beceaf5a1e9929e6cccc9d91eeddf5c65db67dfc70958b0600d72440" exitCode=0 Nov 03 20:21:27.132397 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.132276 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" event={"ID":"81b5ac0a-b306-458b-8b39-96deb3b575fa","Type":"ContainerDied","Data":"89904689beceaf5a1e9929e6cccc9d91eeddf5c65db67dfc70958b0600d72440"} Nov 03 20:21:27.132397 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.132324 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" event={"ID":"81b5ac0a-b306-458b-8b39-96deb3b575fa","Type":"ContainerDied","Data":"e638321011eb8c478ec0de5ee0538ce2a30de935c756553b538456929ba5ac3f"} Nov 03 20:21:27.132397 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.132346 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e638321011eb8c478ec0de5ee0538ce2a30de935c756553b538456929ba5ac3f" Nov 03 20:21:27.135174 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.135142 4457 generic.go:356] "Generic (PLEG): container finished" podID="c254c14f-13f1-4e20-bd27-14585726663f" containerID="8771c19020b4901bc0c96afa8c44d8dc8e7618713784fb46a9a038d75f81e6ad" exitCode=0 Nov 03 20:21:27.135299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.135217 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" event={"ID":"c254c14f-13f1-4e20-bd27-14585726663f","Type":"ContainerDied","Data":"8771c19020b4901bc0c96afa8c44d8dc8e7618713784fb46a9a038d75f81e6ad"} Nov 03 20:21:27.137756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.137704 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" Nov 03 20:21:27.207243 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.207211 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" Nov 03 20:21:27.249260 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.249218 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b5ac0a-b306-458b-8b39-96deb3b575fa-kube-api-access-k6rpn\" (UniqueName: \"kubernetes.io/projected/81b5ac0a-b306-458b-8b39-96deb3b575fa-kube-api-access-k6rpn\") pod \"81b5ac0a-b306-458b-8b39-96deb3b575fa\" (UID: \"81b5ac0a-b306-458b-8b39-96deb3b575fa\") " Nov 03 20:21:27.249503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.249376 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b5ac0a-b306-458b-8b39-96deb3b575fa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b5ac0a-b306-458b-8b39-96deb3b575fa-plugins-config-volume\") pod \"81b5ac0a-b306-458b-8b39-96deb3b575fa\" (UID: \"81b5ac0a-b306-458b-8b39-96deb3b575fa\") " Nov 03 20:21:27.251650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.251616 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b5ac0a-b306-458b-8b39-96deb3b575fa-plugins-config-volume" pod "81b5ac0a-b306-458b-8b39-96deb3b575fa" (UID: "81b5ac0a-b306-458b-8b39-96deb3b575fa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:27.267779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.267741 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b5ac0a-b306-458b-8b39-96deb3b575fa-kube-api-access-k6rpn" pod "81b5ac0a-b306-458b-8b39-96deb3b575fa" (UID: "81b5ac0a-b306-458b-8b39-96deb3b575fa"). InnerVolumeSpecName "kube-api-access-k6rpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:27.352221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.352179 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c254c14f-13f1-4e20-bd27-14585726663f-kube-api-access-9jk86\" (UniqueName: \"kubernetes.io/projected/c254c14f-13f1-4e20-bd27-14585726663f-kube-api-access-9jk86\") pod \"c254c14f-13f1-4e20-bd27-14585726663f\" (UID: \"c254c14f-13f1-4e20-bd27-14585726663f\") " Nov 03 20:21:27.352461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.352248 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c254c14f-13f1-4e20-bd27-14585726663f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c254c14f-13f1-4e20-bd27-14585726663f-plugins-config-volume\") pod \"c254c14f-13f1-4e20-bd27-14585726663f\" (UID: \"c254c14f-13f1-4e20-bd27-14585726663f\") " Nov 03 20:21:27.352679 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.352657 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c254c14f-13f1-4e20-bd27-14585726663f-plugins-config-volume" pod "c254c14f-13f1-4e20-bd27-14585726663f" (UID: "c254c14f-13f1-4e20-bd27-14585726663f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:27.352930 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.352881 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6rpn\" (UniqueName: \"kubernetes.io/projected/81b5ac0a-b306-458b-8b39-96deb3b575fa-kube-api-access-k6rpn\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:27.353010 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.352935 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c254c14f-13f1-4e20-bd27-14585726663f-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:27.353010 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.352952 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81b5ac0a-b306-458b-8b39-96deb3b575fa-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:27.354431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.354407 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c254c14f-13f1-4e20-bd27-14585726663f-kube-api-access-9jk86" pod "c254c14f-13f1-4e20-bd27-14585726663f" (UID: "c254c14f-13f1-4e20-bd27-14585726663f"). InnerVolumeSpecName "kube-api-access-9jk86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:27.455044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:27.454950 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jk86\" (UniqueName: \"kubernetes.io/projected/c254c14f-13f1-4e20-bd27-14585726663f-kube-api-access-9jk86\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:28.153648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:28.150594 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn" Nov 03 20:21:28.157276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:28.157234 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" event={"ID":"c254c14f-13f1-4e20-bd27-14585726663f","Type":"ContainerDied","Data":"c276a343085f2594c755fbc93d2d3d93eeb5d69d5a7d366728f642dac5db23c1"} Nov 03 20:21:28.157505 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:28.157319 4457 scope.go:122] "RemoveContainer" containerID="8771c19020b4901bc0c96afa8c44d8dc8e7618713784fb46a9a038d75f81e6ad" Nov 03 20:21:28.158069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:28.158037 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t" Nov 03 20:21:28.192546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:28.192497 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn"] Nov 03 20:21:28.198695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:28.198655 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-bv6sn"] Nov 03 20:21:28.204082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:28.204053 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t"] Nov 03 20:21:28.234343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:28.234306 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-kkw4t"] Nov 03 20:21:29.760591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:29.760547 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b5ac0a-b306-458b-8b39-96deb3b575fa" path="/var/lib/kubelet/pods/81b5ac0a-b306-458b-8b39-96deb3b575fa/volumes" Nov 03 20:21:29.761157 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:29.761139 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c254c14f-13f1-4e20-bd27-14585726663f" path="/var/lib/kubelet/pods/c254c14f-13f1-4e20-bd27-14585726663f/volumes" Nov 03 20:21:33.421369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.421332 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl"] Nov 03 20:21:33.422150 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422130 4457 cpu_manager.go:414] "containerMap: removing container" podUID="6f0384e1-5813-48bc-9cd4-184ce22ca486" containerName="revoke-keys" Nov 03 20:21:33.422198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422151 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0384e1-5813-48bc-9cd4-184ce22ca486" containerName="revoke-keys" Nov 03 20:21:33.422198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422164 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d1acef2b-8495-41ff-8a07-1797e7a8a1a6" containerName="payload-pre-processing" Nov 03 20:21:33.422198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422170 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1acef2b-8495-41ff-8a07-1797e7a8a1a6" containerName="payload-pre-processing" Nov 03 20:21:33.422198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422194 4457 cpu_manager.go:414] "containerMap: removing container" podUID="81b5ac0a-b306-458b-8b39-96deb3b575fa" containerName="payload-processing" Nov 03 20:21:33.422324 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422199 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b5ac0a-b306-458b-8b39-96deb3b575fa" containerName="payload-processing" Nov 03 20:21:33.422324 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422212 4457 cpu_manager.go:414] "containerMap: removing container" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" Nov 03 20:21:33.422324 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422216 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" Nov 03 20:21:33.422324 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422233 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c254c14f-13f1-4e20-bd27-14585726663f" containerName="payload-pre-processing" Nov 03 20:21:33.422324 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422238 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c254c14f-13f1-4e20-bd27-14585726663f" containerName="payload-pre-processing" Nov 03 20:21:33.422490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422370 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b5ac0a-b306-458b-8b39-96deb3b575fa" containerName="payload-processing" Nov 03 20:21:33.422490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422381 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="27abf71d-0887-4f90-a511-54719c9194db" containerName="istio-proxy" Nov 03 20:21:33.422490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422404 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c254c14f-13f1-4e20-bd27-14585726663f" containerName="payload-pre-processing" Nov 03 20:21:33.422490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422414 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0384e1-5813-48bc-9cd4-184ce22ca486" containerName="revoke-keys" Nov 03 20:21:33.422490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.422421 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1acef2b-8495-41ff-8a07-1797e7a8a1a6" containerName="payload-pre-processing" Nov 03 20:21:33.425815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.425793 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" Nov 03 20:21:33.431201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.430936 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:21:33.431201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.430947 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:21:33.436883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.436859 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl"] Nov 03 20:21:33.483023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.482993 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc"] Nov 03 20:21:33.483530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.483500 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" containerID="cri-o://4023e946742b90a818c32e9a3b61577d9adb17fe2f0e5b34431c2d802b87f940" gracePeriod=30 Nov 03 20:21:33.560409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.560286 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f19886e6-5mdtl\" (UID: \"efe61317-8ea3-451b-ad9a-24678f00034f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" Nov 03 20:21:33.560597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.560463 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f19886e6-5mdtl\" (UID: \"efe61317-8ea3-451b-ad9a-24678f00034f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" Nov 03 20:21:33.560597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.560530 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcbpl\" (UniqueName: \"kubernetes.io/projected/efe61317-8ea3-451b-ad9a-24678f00034f-kube-api-access-lcbpl\") pod \"maas-api-revoke-sub-f19886e6-5mdtl\" (UID: \"efe61317-8ea3-451b-ad9a-24678f00034f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" Nov 03 20:21:33.668409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.667611 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f19886e6-5mdtl\" (UID: \"efe61317-8ea3-451b-ad9a-24678f00034f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" Nov 03 20:21:33.668409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.667676 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f19886e6-5mdtl\" (UID: \"efe61317-8ea3-451b-ad9a-24678f00034f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" Nov 03 20:21:33.668409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.667728 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcbpl\" (UniqueName: \"kubernetes.io/projected/efe61317-8ea3-451b-ad9a-24678f00034f-kube-api-access-lcbpl\") pod \"maas-api-revoke-sub-f19886e6-5mdtl\" (UID: \"efe61317-8ea3-451b-ad9a-24678f00034f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" Nov 03 20:21:33.683174 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.679044 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f19886e6-5mdtl\" (UID: \"efe61317-8ea3-451b-ad9a-24678f00034f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" Nov 03 20:21:33.683174 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.679731 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f19886e6-5mdtl\" (UID: \"efe61317-8ea3-451b-ad9a-24678f00034f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" Nov 03 20:21:33.697410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.692424 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcbpl\" (UniqueName: \"kubernetes.io/projected/efe61317-8ea3-451b-ad9a-24678f00034f-kube-api-access-lcbpl\") pod \"maas-api-revoke-sub-f19886e6-5mdtl\" (UID: \"efe61317-8ea3-451b-ad9a-24678f00034f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" Nov 03 20:21:33.789413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:33.789155 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" Nov 03 20:21:34.174021 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:21:34.173965 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefe61317_8ea3_451b_ad9a_24678f00034f.slice/crio-0d14e717dfa3fb1f53c49b89724c94a8368496c0af5aeaa3883ab44c67827e81 WatchSource:0}: Error finding container 0d14e717dfa3fb1f53c49b89724c94a8368496c0af5aeaa3883ab44c67827e81: Status 404 returned error can't find the container with id 0d14e717dfa3fb1f53c49b89724c94a8368496c0af5aeaa3883ab44c67827e81 Nov 03 20:21:34.175202 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:34.175181 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl"] Nov 03 20:21:34.234162 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:34.234124 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" event={"ID":"efe61317-8ea3-451b-ad9a-24678f00034f","Type":"ContainerStarted","Data":"0d14e717dfa3fb1f53c49b89724c94a8368496c0af5aeaa3883ab44c67827e81"} Nov 03 20:21:35.402335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:35.402279 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" event={"ID":"efe61317-8ea3-451b-ad9a-24678f00034f","Type":"ContainerStarted","Data":"049e63fdec50efe114a7882d06eddf85dc533a788c3eea86551bbbac62749c91"} Nov 03 20:21:35.498466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:35.488488 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" podStartSLOduration=2.488466824 podStartE2EDuration="2.488466824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:35.487032963 +0000 UTC m=+1473.862629711" watchObservedRunningTime="2026-11-03 20:21:35.488466824 +0000 UTC m=+1473.864063571" Nov 03 20:21:36.148545 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:21:36.148507 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:21:36.416556 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:36.416456 4457 generic.go:356] "Generic (PLEG): container finished" podID="efe61317-8ea3-451b-ad9a-24678f00034f" containerID="049e63fdec50efe114a7882d06eddf85dc533a788c3eea86551bbbac62749c91" exitCode=0 Nov 03 20:21:36.417032 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:36.416776 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" event={"ID":"efe61317-8ea3-451b-ad9a-24678f00034f","Type":"ContainerDied","Data":"049e63fdec50efe114a7882d06eddf85dc533a788c3eea86551bbbac62749c91"} Nov 03 20:21:37.691560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:37.691487 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" Nov 03 20:21:37.853509 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:37.853472 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-openshift-ca\") pod \"efe61317-8ea3-451b-ad9a-24678f00034f\" (UID: \"efe61317-8ea3-451b-ad9a-24678f00034f\") " Nov 03 20:21:37.853740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:37.853719 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efe61317-8ea3-451b-ad9a-24678f00034f-kube-api-access-lcbpl\" (UniqueName: \"kubernetes.io/projected/efe61317-8ea3-451b-ad9a-24678f00034f-kube-api-access-lcbpl\") pod \"efe61317-8ea3-451b-ad9a-24678f00034f\" (UID: \"efe61317-8ea3-451b-ad9a-24678f00034f\") " Nov 03 20:21:37.853831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:37.853778 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-xks-ca\") pod \"efe61317-8ea3-451b-ad9a-24678f00034f\" (UID: \"efe61317-8ea3-451b-ad9a-24678f00034f\") " Nov 03 20:21:37.853831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:37.853788 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-openshift-ca" pod "efe61317-8ea3-451b-ad9a-24678f00034f" (UID: "efe61317-8ea3-451b-ad9a-24678f00034f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:37.854362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:37.854341 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:37.855858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:37.855825 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-xks-ca" pod "efe61317-8ea3-451b-ad9a-24678f00034f" (UID: "efe61317-8ea3-451b-ad9a-24678f00034f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:21:37.865888 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:37.865855 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efe61317-8ea3-451b-ad9a-24678f00034f-kube-api-access-lcbpl" pod "efe61317-8ea3-451b-ad9a-24678f00034f" (UID: "efe61317-8ea3-451b-ad9a-24678f00034f"). InnerVolumeSpecName "kube-api-access-lcbpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:37.955473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:37.955357 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcbpl\" (UniqueName: \"kubernetes.io/projected/efe61317-8ea3-451b-ad9a-24678f00034f-kube-api-access-lcbpl\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:37.955473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:37.955408 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efe61317-8ea3-451b-ad9a-24678f00034f-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:38.437042 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.437002 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" Nov 03 20:21:38.437220 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.437008 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl" event={"ID":"efe61317-8ea3-451b-ad9a-24678f00034f","Type":"ContainerDied","Data":"0d14e717dfa3fb1f53c49b89724c94a8368496c0af5aeaa3883ab44c67827e81"} Nov 03 20:21:38.437220 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.437133 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d14e717dfa3fb1f53c49b89724c94a8368496c0af5aeaa3883ab44c67827e81" Nov 03 20:21:38.783507 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.783479 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:21:38.873717 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.873679 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c79f72-431b-4137-950f-b5f8d68f9a9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c79f72-431b-4137-950f-b5f8d68f9a9e-istiod-ca-cert\") pod \"09c79f72-431b-4137-950f-b5f8d68f9a9e\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " Nov 03 20:21:38.874120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.874093 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c79f72-431b-4137-950f-b5f8d68f9a9e-istiod-ca-cert" pod "09c79f72-431b-4137-950f-b5f8d68f9a9e" (UID: "09c79f72-431b-4137-950f-b5f8d68f9a9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:38.874237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.874220 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-credential-socket\") pod \"09c79f72-431b-4137-950f-b5f8d68f9a9e\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " Nov 03 20:21:38.874904 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.874875 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-credential-socket" pod "09c79f72-431b-4137-950f-b5f8d68f9a9e" (UID: "09c79f72-431b-4137-950f-b5f8d68f9a9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:38.875031 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.874919 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-token\" (UniqueName: \"kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-token\") pod \"09c79f72-431b-4137-950f-b5f8d68f9a9e\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " Nov 03 20:21:38.875087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.875045 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-envoy\") pod \"09c79f72-431b-4137-950f-b5f8d68f9a9e\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " Nov 03 20:21:38.875154 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.875106 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-podinfo\") pod \"09c79f72-431b-4137-950f-b5f8d68f9a9e\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " Nov 03 20:21:38.875229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.875210 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-kube-api-access-wxq4c\" (UniqueName: \"kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-kube-api-access-wxq4c\") pod \"09c79f72-431b-4137-950f-b5f8d68f9a9e\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " Nov 03 20:21:38.875284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.875265 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-data\") pod \"09c79f72-431b-4137-950f-b5f8d68f9a9e\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " Nov 03 20:21:38.875334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.875321 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-socket\") pod \"09c79f72-431b-4137-950f-b5f8d68f9a9e\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " Nov 03 20:21:38.875403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.875363 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-certs\") pod \"09c79f72-431b-4137-950f-b5f8d68f9a9e\" (UID: \"09c79f72-431b-4137-950f-b5f8d68f9a9e\") " Nov 03 20:21:38.876360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.876337 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c79f72-431b-4137-950f-b5f8d68f9a9e-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:38.876473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.876364 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:38.877241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.877212 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-token" pod "09c79f72-431b-4137-950f-b5f8d68f9a9e" (UID: "09c79f72-431b-4137-950f-b5f8d68f9a9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:38.877496 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.877470 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-socket" pod "09c79f72-431b-4137-950f-b5f8d68f9a9e" (UID: "09c79f72-431b-4137-950f-b5f8d68f9a9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:38.879971 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.879929 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-podinfo" pod "09c79f72-431b-4137-950f-b5f8d68f9a9e" (UID: "09c79f72-431b-4137-950f-b5f8d68f9a9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:38.880098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.880074 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-certs" pod "09c79f72-431b-4137-950f-b5f8d68f9a9e" (UID: "09c79f72-431b-4137-950f-b5f8d68f9a9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:38.880414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.880367 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-envoy" pod "09c79f72-431b-4137-950f-b5f8d68f9a9e" (UID: "09c79f72-431b-4137-950f-b5f8d68f9a9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:38.881930 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.881901 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-data" pod "09c79f72-431b-4137-950f-b5f8d68f9a9e" (UID: "09c79f72-431b-4137-950f-b5f8d68f9a9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:38.882338 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.882314 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-kube-api-access-wxq4c" pod "09c79f72-431b-4137-950f-b5f8d68f9a9e" (UID: "09c79f72-431b-4137-950f-b5f8d68f9a9e"). InnerVolumeSpecName "kube-api-access-wxq4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:38.977749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.977714 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:38.977749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.977743 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:38.977749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.977752 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:38.977749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.977760 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:38.978092 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.977769 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:38.978092 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.977778 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxq4c\" (UniqueName: \"kubernetes.io/projected/09c79f72-431b-4137-950f-b5f8d68f9a9e-kube-api-access-wxq4c\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:38.978092 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:38.977786 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c79f72-431b-4137-950f-b5f8d68f9a9e-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:39.448288 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.448252 4457 generic.go:356] "Generic (PLEG): container finished" podID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerID="4023e946742b90a818c32e9a3b61577d9adb17fe2f0e5b34431c2d802b87f940" exitCode=0 Nov 03 20:21:39.448531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.448310 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" Nov 03 20:21:39.448531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.448328 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" event={"ID":"09c79f72-431b-4137-950f-b5f8d68f9a9e","Type":"ContainerDied","Data":"4023e946742b90a818c32e9a3b61577d9adb17fe2f0e5b34431c2d802b87f940"} Nov 03 20:21:39.448531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.448412 4457 scope.go:122] "RemoveContainer" containerID="4023e946742b90a818c32e9a3b61577d9adb17fe2f0e5b34431c2d802b87f940" Nov 03 20:21:39.448531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.448427 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc" event={"ID":"09c79f72-431b-4137-950f-b5f8d68f9a9e","Type":"ContainerDied","Data":"a1c8ad50b069bd6dfa9cde96ee1bcfeeae4c68f8f5cbfe41e03fcccd5cc446c4"} Nov 03 20:21:39.463725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.463700 4457 scope.go:122] "RemoveContainer" containerID="94d8811fd536c2ba120cffa921554030600eeed3a885381a00c87370091c9fe1" Nov 03 20:21:39.475968 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.475941 4457 scope.go:122] "RemoveContainer" containerID="4023e946742b90a818c32e9a3b61577d9adb17fe2f0e5b34431c2d802b87f940" Nov 03 20:21:39.476455 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:21:39.476379 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4023e946742b90a818c32e9a3b61577d9adb17fe2f0e5b34431c2d802b87f940\": container with ID starting with 4023e946742b90a818c32e9a3b61577d9adb17fe2f0e5b34431c2d802b87f940 not found: ID does not exist" containerID="4023e946742b90a818c32e9a3b61577d9adb17fe2f0e5b34431c2d802b87f940" Nov 03 20:21:39.476563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.476465 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4023e946742b90a818c32e9a3b61577d9adb17fe2f0e5b34431c2d802b87f940"} err="failed to get container status \"4023e946742b90a818c32e9a3b61577d9adb17fe2f0e5b34431c2d802b87f940\": rpc error: code = NotFound desc = could not find container \"4023e946742b90a818c32e9a3b61577d9adb17fe2f0e5b34431c2d802b87f940\": container with ID starting with 4023e946742b90a818c32e9a3b61577d9adb17fe2f0e5b34431c2d802b87f940 not found: ID does not exist" Nov 03 20:21:39.476563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.476492 4457 scope.go:122] "RemoveContainer" containerID="94d8811fd536c2ba120cffa921554030600eeed3a885381a00c87370091c9fe1" Nov 03 20:21:39.477284 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:21:39.477263 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d8811fd536c2ba120cffa921554030600eeed3a885381a00c87370091c9fe1\": container with ID starting with 94d8811fd536c2ba120cffa921554030600eeed3a885381a00c87370091c9fe1 not found: ID does not exist" containerID="94d8811fd536c2ba120cffa921554030600eeed3a885381a00c87370091c9fe1" Nov 03 20:21:39.477405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.477289 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d8811fd536c2ba120cffa921554030600eeed3a885381a00c87370091c9fe1"} err="failed to get container status \"94d8811fd536c2ba120cffa921554030600eeed3a885381a00c87370091c9fe1\": rpc error: code = NotFound desc = could not find container \"94d8811fd536c2ba120cffa921554030600eeed3a885381a00c87370091c9fe1\": container with ID starting with 94d8811fd536c2ba120cffa921554030600eeed3a885381a00c87370091c9fe1 not found: ID does not exist" Nov 03 20:21:39.483655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.483630 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc"] Nov 03 20:21:39.494247 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.494212 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9af0cd-openshift-default-76698cc64f-69glc"] Nov 03 20:21:39.602026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.601994 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz"] Nov 03 20:21:39.602784 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.602758 4457 cpu_manager.go:414] "containerMap: removing container" podUID="efe61317-8ea3-451b-ad9a-24678f00034f" containerName="revoke-keys" Nov 03 20:21:39.602893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.602785 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efe61317-8ea3-451b-ad9a-24678f00034f" containerName="revoke-keys" Nov 03 20:21:39.602893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.602842 4457 cpu_manager.go:414] "containerMap: removing container" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" Nov 03 20:21:39.602893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.602851 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" Nov 03 20:21:39.602893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.602881 4457 cpu_manager.go:414] "containerMap: removing container" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" Nov 03 20:21:39.602893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.602887 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" Nov 03 20:21:39.603141 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.603125 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" Nov 03 20:21:39.603175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.603154 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" containerName="istio-proxy" Nov 03 20:21:39.603204 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.603173 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="efe61317-8ea3-451b-ad9a-24678f00034f" containerName="revoke-keys" Nov 03 20:21:39.606818 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.606800 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" Nov 03 20:21:39.619509 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.619485 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6cf8ae\"" Nov 03 20:21:39.622311 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.622284 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz"] Nov 03 20:21:39.627327 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.627301 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff"] Nov 03 20:21:39.637490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.637461 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" Nov 03 20:21:39.653613 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.653464 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6cf8ae-dockercfg-zdcnw\"" Nov 03 20:21:39.655723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.655679 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff"] Nov 03 20:21:39.691403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.691354 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f934e95-c712-48af-83d4-a7b80d870089-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff\" (UID: \"7f934e95-c712-48af-83d4-a7b80d870089\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" Nov 03 20:21:39.691565 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.691498 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cch84\" (UniqueName: \"kubernetes.io/projected/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-kube-api-access-cch84\") pod \"payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz\" (UID: \"9deb1243-b43c-4e37-94a9-1eb5e96c21cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" Nov 03 20:21:39.691635 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.691587 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2d8j\" (UniqueName: \"kubernetes.io/projected/7f934e95-c712-48af-83d4-a7b80d870089-kube-api-access-z2d8j\") pod \"payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff\" (UID: \"7f934e95-c712-48af-83d4-a7b80d870089\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" Nov 03 20:21:39.691856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.691836 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz\" (UID: \"9deb1243-b43c-4e37-94a9-1eb5e96c21cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" Nov 03 20:21:39.794201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.794111 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cch84\" (UniqueName: \"kubernetes.io/projected/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-kube-api-access-cch84\") pod \"payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz\" (UID: \"9deb1243-b43c-4e37-94a9-1eb5e96c21cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" Nov 03 20:21:39.803503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.803464 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2d8j\" (UniqueName: \"kubernetes.io/projected/7f934e95-c712-48af-83d4-a7b80d870089-kube-api-access-z2d8j\") pod \"payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff\" (UID: \"7f934e95-c712-48af-83d4-a7b80d870089\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" Nov 03 20:21:39.804298 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.804272 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz\" (UID: \"9deb1243-b43c-4e37-94a9-1eb5e96c21cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" Nov 03 20:21:39.805562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.805433 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz\" (UID: \"9deb1243-b43c-4e37-94a9-1eb5e96c21cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" Nov 03 20:21:39.805766 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.805746 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f934e95-c712-48af-83d4-a7b80d870089-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff\" (UID: \"7f934e95-c712-48af-83d4-a7b80d870089\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" Nov 03 20:21:39.820573 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.806591 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f934e95-c712-48af-83d4-a7b80d870089-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff\" (UID: \"7f934e95-c712-48af-83d4-a7b80d870089\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" Nov 03 20:21:39.841268 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.841209 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cch84\" (UniqueName: \"kubernetes.io/projected/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-kube-api-access-cch84\") pod \"payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz\" (UID: \"9deb1243-b43c-4e37-94a9-1eb5e96c21cc\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" Nov 03 20:21:39.852476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.848053 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2d8j\" (UniqueName: \"kubernetes.io/projected/7f934e95-c712-48af-83d4-a7b80d870089-kube-api-access-z2d8j\") pod \"payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff\" (UID: \"7f934e95-c712-48af-83d4-a7b80d870089\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" Nov 03 20:21:39.896464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.880182 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c79f72-431b-4137-950f-b5f8d68f9a9e" path="/var/lib/kubelet/pods/09c79f72-431b-4137-950f-b5f8d68f9a9e/volumes" Nov 03 20:21:39.974410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.955906 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" Nov 03 20:21:39.974410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:39.968174 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" Nov 03 20:21:40.526070 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:40.521883 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff"] Nov 03 20:21:40.526070 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:40.523236 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz"] Nov 03 20:21:40.537452 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:21:40.537415 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f934e95_c712_48af_83d4_a7b80d870089.slice/crio-3f04d133e980627250d3d0892c7f5c806431c0f094862fe34a36c7aa7236e8ef WatchSource:0}: Error finding container 3f04d133e980627250d3d0892c7f5c806431c0f094862fe34a36c7aa7236e8ef: Status 404 returned error can't find the container with id 3f04d133e980627250d3d0892c7f5c806431c0f094862fe34a36c7aa7236e8ef Nov 03 20:21:40.538522 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:21:40.538496 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9deb1243_b43c_4e37_94a9_1eb5e96c21cc.slice/crio-4df0289f267fca242e9e1c9ec46757cbc4db0afce006f943f9da3497eb5e70d2 WatchSource:0}: Error finding container 4df0289f267fca242e9e1c9ec46757cbc4db0afce006f943f9da3497eb5e70d2: Status 404 returned error can't find the container with id 4df0289f267fca242e9e1c9ec46757cbc4db0afce006f943f9da3497eb5e70d2 Nov 03 20:21:41.552023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:41.551989 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" event={"ID":"7f934e95-c712-48af-83d4-a7b80d870089","Type":"ContainerStarted","Data":"769b176777ba6c1539ce2059ff6006c048d1e64dc330e477bb939645bb5ad55f"} Nov 03 20:21:41.552493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:41.552051 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" event={"ID":"7f934e95-c712-48af-83d4-a7b80d870089","Type":"ContainerStarted","Data":"3f04d133e980627250d3d0892c7f5c806431c0f094862fe34a36c7aa7236e8ef"} Nov 03 20:21:41.552493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:41.552083 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" Nov 03 20:21:41.554134 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:41.554101 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" event={"ID":"9deb1243-b43c-4e37-94a9-1eb5e96c21cc","Type":"ContainerStarted","Data":"9f3e9cf7c85c34eab1eb05a24c462671d4f5414f348d682ef5595d83e6572195"} Nov 03 20:21:41.554249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:41.554159 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" event={"ID":"9deb1243-b43c-4e37-94a9-1eb5e96c21cc","Type":"ContainerStarted","Data":"4df0289f267fca242e9e1c9ec46757cbc4db0afce006f943f9da3497eb5e70d2"} Nov 03 20:21:41.554249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:41.554189 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" Nov 03 20:21:41.575052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:41.574998 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" podStartSLOduration=2.574983644 podStartE2EDuration="2.574983644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:41.569451616 +0000 UTC m=+1479.945048364" watchObservedRunningTime="2026-11-03 20:21:41.574983644 +0000 UTC m=+1479.950580420" Nov 03 20:21:41.598943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:41.598875 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" podStartSLOduration=2.598853885 podStartE2EDuration="2.598853885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:41.595720554 +0000 UTC m=+1479.971317323" watchObservedRunningTime="2026-11-03 20:21:41.598853885 +0000 UTC m=+1479.974450632" Nov 03 20:21:43.619354 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.612801 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4"] Nov 03 20:21:43.626359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.626331 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4"] Nov 03 20:21:43.626696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.626678 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" Nov 03 20:21:43.654894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.654859 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:21:43.664513 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.655583 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:21:43.673922 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.673885 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426daf06-f207-404a-8361-395cb6745958-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4\" (UID: \"426daf06-f207-404a-8361-395cb6745958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" Nov 03 20:21:43.676630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.676598 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v84cl\" (UniqueName: \"kubernetes.io/projected/426daf06-f207-404a-8361-395cb6745958-kube-api-access-v84cl\") pod \"maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4\" (UID: \"426daf06-f207-404a-8361-395cb6745958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" Nov 03 20:21:43.677183 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.677151 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426daf06-f207-404a-8361-395cb6745958-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4\" (UID: \"426daf06-f207-404a-8361-395cb6745958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" Nov 03 20:21:43.778543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.778510 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426daf06-f207-404a-8361-395cb6745958-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4\" (UID: \"426daf06-f207-404a-8361-395cb6745958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" Nov 03 20:21:43.778725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.778653 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426daf06-f207-404a-8361-395cb6745958-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4\" (UID: \"426daf06-f207-404a-8361-395cb6745958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" Nov 03 20:21:43.778768 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.778723 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v84cl\" (UniqueName: \"kubernetes.io/projected/426daf06-f207-404a-8361-395cb6745958-kube-api-access-v84cl\") pod \"maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4\" (UID: \"426daf06-f207-404a-8361-395cb6745958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" Nov 03 20:21:43.779339 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.779314 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426daf06-f207-404a-8361-395cb6745958-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4\" (UID: \"426daf06-f207-404a-8361-395cb6745958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" Nov 03 20:21:43.781127 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.781103 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426daf06-f207-404a-8361-395cb6745958-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4\" (UID: \"426daf06-f207-404a-8361-395cb6745958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" Nov 03 20:21:43.793825 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.793799 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v84cl\" (UniqueName: \"kubernetes.io/projected/426daf06-f207-404a-8361-395cb6745958-kube-api-access-v84cl\") pod \"maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4\" (UID: \"426daf06-f207-404a-8361-395cb6745958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" Nov 03 20:21:43.992845 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:43.992705 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" Nov 03 20:21:44.482625 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:44.482528 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4"] Nov 03 20:21:44.486163 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:21:44.486141 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426daf06_f207_404a_8361_395cb6745958.slice/crio-8beb9f0c9ee707e42bbfd78b69d8b4379417e2bf761e50d0dcd3f4874b09ac0c WatchSource:0}: Error finding container 8beb9f0c9ee707e42bbfd78b69d8b4379417e2bf761e50d0dcd3f4874b09ac0c: Status 404 returned error can't find the container with id 8beb9f0c9ee707e42bbfd78b69d8b4379417e2bf761e50d0dcd3f4874b09ac0c Nov 03 20:21:44.610240 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:44.610199 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" event={"ID":"426daf06-f207-404a-8361-395cb6745958","Type":"ContainerStarted","Data":"8beb9f0c9ee707e42bbfd78b69d8b4379417e2bf761e50d0dcd3f4874b09ac0c"} Nov 03 20:21:45.623837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:45.623799 4457 generic.go:356] "Generic (PLEG): container finished" podID="426daf06-f207-404a-8361-395cb6745958" containerID="3e316fea2beaeb3010b82f51ea1d006251f9e94269731070754928b6d80c3322" exitCode=0 Nov 03 20:21:45.624292 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:45.623892 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" event={"ID":"426daf06-f207-404a-8361-395cb6745958","Type":"ContainerDied","Data":"3e316fea2beaeb3010b82f51ea1d006251f9e94269731070754928b6d80c3322"} Nov 03 20:21:46.290183 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.290132 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7"] Nov 03 20:21:46.295067 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.295039 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" Nov 03 20:21:46.305721 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.305694 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-683ba6-dockercfg-fp2vz\"" Nov 03 20:21:46.305967 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.305944 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-683ba6\"" Nov 03 20:21:46.307583 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.307557 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7"] Nov 03 20:21:46.326348 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.326315 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc"] Nov 03 20:21:46.332325 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.332299 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" Nov 03 20:21:46.341429 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.341401 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc"] Nov 03 20:21:46.347851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.347817 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f09f12d-25df-412a-8090-101261a0d96e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7\" (UID: \"0f09f12d-25df-412a-8090-101261a0d96e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" Nov 03 20:21:46.348101 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.347903 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc\" (UID: \"81847f97-4b4e-460a-9a37-6ea2e76bb1d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" Nov 03 20:21:46.348101 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.348055 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94p7l\" (UniqueName: \"kubernetes.io/projected/0f09f12d-25df-412a-8090-101261a0d96e-kube-api-access-94p7l\") pod \"payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7\" (UID: \"0f09f12d-25df-412a-8090-101261a0d96e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" Nov 03 20:21:46.348369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.348165 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46vrl\" (UniqueName: \"kubernetes.io/projected/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-kube-api-access-46vrl\") pod \"payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc\" (UID: \"81847f97-4b4e-460a-9a37-6ea2e76bb1d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" Nov 03 20:21:46.452410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.451437 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94p7l\" (UniqueName: \"kubernetes.io/projected/0f09f12d-25df-412a-8090-101261a0d96e-kube-api-access-94p7l\") pod \"payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7\" (UID: \"0f09f12d-25df-412a-8090-101261a0d96e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" Nov 03 20:21:46.452410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.451588 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46vrl\" (UniqueName: \"kubernetes.io/projected/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-kube-api-access-46vrl\") pod \"payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc\" (UID: \"81847f97-4b4e-460a-9a37-6ea2e76bb1d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" Nov 03 20:21:46.452410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.451717 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f09f12d-25df-412a-8090-101261a0d96e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7\" (UID: \"0f09f12d-25df-412a-8090-101261a0d96e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" Nov 03 20:21:46.452410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.451782 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc\" (UID: \"81847f97-4b4e-460a-9a37-6ea2e76bb1d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" Nov 03 20:21:46.466343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.452913 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc\" (UID: \"81847f97-4b4e-460a-9a37-6ea2e76bb1d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" Nov 03 20:21:46.466343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.454183 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f09f12d-25df-412a-8090-101261a0d96e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7\" (UID: \"0f09f12d-25df-412a-8090-101261a0d96e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" Nov 03 20:21:46.699540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.699458 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94p7l\" (UniqueName: \"kubernetes.io/projected/0f09f12d-25df-412a-8090-101261a0d96e-kube-api-access-94p7l\") pod \"payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7\" (UID: \"0f09f12d-25df-412a-8090-101261a0d96e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" Nov 03 20:21:46.702038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.702013 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46vrl\" (UniqueName: \"kubernetes.io/projected/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-kube-api-access-46vrl\") pod \"payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc\" (UID: \"81847f97-4b4e-460a-9a37-6ea2e76bb1d4\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" Nov 03 20:21:46.914175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.914143 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" Nov 03 20:21:46.918265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.918233 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" Nov 03 20:21:46.921714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.921682 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/426daf06-f207-404a-8361-395cb6745958-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426daf06-f207-404a-8361-395cb6745958-maas-api-xks-ca\") pod \"426daf06-f207-404a-8361-395cb6745958\" (UID: \"426daf06-f207-404a-8361-395cb6745958\") " Nov 03 20:21:46.921838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.921758 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426daf06-f207-404a-8361-395cb6745958-kube-api-access-v84cl\" (UniqueName: \"kubernetes.io/projected/426daf06-f207-404a-8361-395cb6745958-kube-api-access-v84cl\") pod \"426daf06-f207-404a-8361-395cb6745958\" (UID: \"426daf06-f207-404a-8361-395cb6745958\") " Nov 03 20:21:46.921913 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.921896 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426daf06-f207-404a-8361-395cb6745958-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426daf06-f207-404a-8361-395cb6745958-maas-api-openshift-ca\") pod \"426daf06-f207-404a-8361-395cb6745958\" (UID: \"426daf06-f207-404a-8361-395cb6745958\") " Nov 03 20:21:46.922250 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.922224 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426daf06-f207-404a-8361-395cb6745958-maas-api-openshift-ca" pod "426daf06-f207-404a-8361-395cb6745958" (UID: "426daf06-f207-404a-8361-395cb6745958"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:46.922626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.922602 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/426daf06-f207-404a-8361-395cb6745958-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:46.923989 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.923969 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426daf06-f207-404a-8361-395cb6745958-maas-api-xks-ca" pod "426daf06-f207-404a-8361-395cb6745958" (UID: "426daf06-f207-404a-8361-395cb6745958"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:21:46.924278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.924262 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426daf06-f207-404a-8361-395cb6745958-kube-api-access-v84cl" pod "426daf06-f207-404a-8361-395cb6745958" (UID: "426daf06-f207-404a-8361-395cb6745958"). InnerVolumeSpecName "kube-api-access-v84cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:46.962353 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:46.962084 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" Nov 03 20:21:47.025926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.024661 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/426daf06-f207-404a-8361-395cb6745958-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:47.025926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.024698 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v84cl\" (UniqueName: \"kubernetes.io/projected/426daf06-f207-404a-8361-395cb6745958-kube-api-access-v84cl\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:47.261145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.261113 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7"] Nov 03 20:21:47.264976 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:21:47.264944 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f09f12d_25df_412a_8090_101261a0d96e.slice/crio-5fdd29c8bb38631fe2a0e49293aa18ffde7a8b5f615562858d07e8ff99cc3b08 WatchSource:0}: Error finding container 5fdd29c8bb38631fe2a0e49293aa18ffde7a8b5f615562858d07e8ff99cc3b08: Status 404 returned error can't find the container with id 5fdd29c8bb38631fe2a0e49293aa18ffde7a8b5f615562858d07e8ff99cc3b08 Nov 03 20:21:47.320113 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.320086 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8"] Nov 03 20:21:47.320397 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.320332 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" containerID="cri-o://273d86fa3f9797ed1a1b5109dc913dbb73f8b93b8bebdffe6938fe0e2236ca4e" gracePeriod=30 Nov 03 20:21:47.382350 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.382314 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc"] Nov 03 20:21:47.686078 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.686050 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" Nov 03 20:21:47.686732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.686700 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4" event={"ID":"426daf06-f207-404a-8361-395cb6745958","Type":"ContainerDied","Data":"8beb9f0c9ee707e42bbfd78b69d8b4379417e2bf761e50d0dcd3f4874b09ac0c"} Nov 03 20:21:47.686944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.686927 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8beb9f0c9ee707e42bbfd78b69d8b4379417e2bf761e50d0dcd3f4874b09ac0c" Nov 03 20:21:47.689074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.689047 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" event={"ID":"81847f97-4b4e-460a-9a37-6ea2e76bb1d4","Type":"ContainerStarted","Data":"1e88280b432bade611b5a1bd57645b5e06b426a3ba9e0e0e7030d22a11a62f24"} Nov 03 20:21:47.689178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.689099 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" event={"ID":"81847f97-4b4e-460a-9a37-6ea2e76bb1d4","Type":"ContainerStarted","Data":"40530a0c97dbc6dd7c7e6d73255dd331ad1e1427b8707fc81d0e7497d68cd094"} Nov 03 20:21:47.690131 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.690112 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" Nov 03 20:21:47.692265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.692239 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" event={"ID":"0f09f12d-25df-412a-8090-101261a0d96e","Type":"ContainerStarted","Data":"6922c1f140ca25466434dd441b7139dad872aeaa61c8300cb33ddd883f840314"} Nov 03 20:21:47.692374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.692288 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" event={"ID":"0f09f12d-25df-412a-8090-101261a0d96e","Type":"ContainerStarted","Data":"5fdd29c8bb38631fe2a0e49293aa18ffde7a8b5f615562858d07e8ff99cc3b08"} Nov 03 20:21:47.693084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.693063 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" Nov 03 20:21:47.710540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.710495 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" podStartSLOduration=1.7104824220000001 podStartE2EDuration="1.710482422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:47.708015042 +0000 UTC m=+1486.083611790" watchObservedRunningTime="2026-11-03 20:21:47.710482422 +0000 UTC m=+1486.086079168" Nov 03 20:21:47.733952 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:47.733905 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" podStartSLOduration=1.733892795 podStartE2EDuration="1.733892795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:21:47.727719137 +0000 UTC m=+1486.103315884" watchObservedRunningTime="2026-11-03 20:21:47.733892795 +0000 UTC m=+1486.109489584" Nov 03 20:21:51.808943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:51.808909 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb"] Nov 03 20:21:51.821735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:51.821701 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-z9wkb"] Nov 03 20:21:52.570492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:52.570461 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" Nov 03 20:21:52.571211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:52.571182 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" Nov 03 20:21:52.766551 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:52.766459 4457 generic.go:356] "Generic (PLEG): container finished" podID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerID="273d86fa3f9797ed1a1b5109dc913dbb73f8b93b8bebdffe6938fe0e2236ca4e" exitCode=0 Nov 03 20:21:52.766727 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:52.766539 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" event={"ID":"2690f13c-4a95-48d3-9b47-598b87d5d992","Type":"ContainerDied","Data":"273d86fa3f9797ed1a1b5109dc913dbb73f8b93b8bebdffe6938fe0e2236ca4e"} Nov 03 20:21:53.069064 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.069029 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:21:53.198763 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.198722 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-token\" (UniqueName: \"kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-token\") pod \"2690f13c-4a95-48d3-9b47-598b87d5d992\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " Nov 03 20:21:53.198932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.198815 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-data\") pod \"2690f13c-4a95-48d3-9b47-598b87d5d992\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " Nov 03 20:21:53.199021 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.198949 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-kube-api-access-r8tj8\" (UniqueName: \"kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-kube-api-access-r8tj8\") pod \"2690f13c-4a95-48d3-9b47-598b87d5d992\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " Nov 03 20:21:53.199021 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.198993 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-certs\") pod \"2690f13c-4a95-48d3-9b47-598b87d5d992\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " Nov 03 20:21:53.199124 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.199071 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-socket\") pod \"2690f13c-4a95-48d3-9b47-598b87d5d992\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " Nov 03 20:21:53.199240 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.199197 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2690f13c-4a95-48d3-9b47-598b87d5d992-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2690f13c-4a95-48d3-9b47-598b87d5d992-istiod-ca-cert\") pod \"2690f13c-4a95-48d3-9b47-598b87d5d992\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " Nov 03 20:21:53.199508 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.199374 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-podinfo\") pod \"2690f13c-4a95-48d3-9b47-598b87d5d992\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " Nov 03 20:21:53.201161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.200566 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-certs" pod "2690f13c-4a95-48d3-9b47-598b87d5d992" (UID: "2690f13c-4a95-48d3-9b47-598b87d5d992"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.201267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.201177 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-data" pod "2690f13c-4a95-48d3-9b47-598b87d5d992" (UID: "2690f13c-4a95-48d3-9b47-598b87d5d992"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.201805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.201782 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-socket" pod "2690f13c-4a95-48d3-9b47-598b87d5d992" (UID: "2690f13c-4a95-48d3-9b47-598b87d5d992"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.201960 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.201908 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2690f13c-4a95-48d3-9b47-598b87d5d992-istiod-ca-cert" pod "2690f13c-4a95-48d3-9b47-598b87d5d992" (UID: "2690f13c-4a95-48d3-9b47-598b87d5d992"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:53.202086 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.202063 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-credential-socket\") pod \"2690f13c-4a95-48d3-9b47-598b87d5d992\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " Nov 03 20:21:53.202175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.202157 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-envoy\") pod \"2690f13c-4a95-48d3-9b47-598b87d5d992\" (UID: \"2690f13c-4a95-48d3-9b47-598b87d5d992\") " Nov 03 20:21:53.202341 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.202322 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-credential-socket" pod "2690f13c-4a95-48d3-9b47-598b87d5d992" (UID: "2690f13c-4a95-48d3-9b47-598b87d5d992"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.206675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.206037 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.206675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.206066 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.206675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.206081 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.206675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.206097 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2690f13c-4a95-48d3-9b47-598b87d5d992-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.206675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.206113 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.208730 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.208665 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-podinfo" pod "2690f13c-4a95-48d3-9b47-598b87d5d992" (UID: "2690f13c-4a95-48d3-9b47-598b87d5d992"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:21:53.208730 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.208717 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-envoy" pod "2690f13c-4a95-48d3-9b47-598b87d5d992" (UID: "2690f13c-4a95-48d3-9b47-598b87d5d992"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:21:53.209334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.209301 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-kube-api-access-r8tj8" pod "2690f13c-4a95-48d3-9b47-598b87d5d992" (UID: "2690f13c-4a95-48d3-9b47-598b87d5d992"). InnerVolumeSpecName "kube-api-access-r8tj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:53.212024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.211994 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-token" pod "2690f13c-4a95-48d3-9b47-598b87d5d992" (UID: "2690f13c-4a95-48d3-9b47-598b87d5d992"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:53.307961 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.307926 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.307961 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.307957 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.307961 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.307967 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.308200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.307977 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8tj8\" (UniqueName: \"kubernetes.io/projected/2690f13c-4a95-48d3-9b47-598b87d5d992-kube-api-access-r8tj8\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:53.761078 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.760977 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0384e1-5813-48bc-9cd4-184ce22ca486" path="/var/lib/kubelet/pods/6f0384e1-5813-48bc-9cd4-184ce22ca486/volumes" Nov 03 20:21:53.787012 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.786981 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" Nov 03 20:21:53.787012 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.786997 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8" event={"ID":"2690f13c-4a95-48d3-9b47-598b87d5d992","Type":"ContainerDied","Data":"34621d62042da4e2c0d9bc1aabf76ffc067a7b8bc63bf86bc507ed5a91b96942"} Nov 03 20:21:53.787229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.787056 4457 scope.go:122] "RemoveContainer" containerID="273d86fa3f9797ed1a1b5109dc913dbb73f8b93b8bebdffe6938fe0e2236ca4e" Nov 03 20:21:53.811357 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.811296 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8"] Nov 03 20:21:53.816479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.816441 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-zszr8"] Nov 03 20:21:53.862730 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:53.862690 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws"] Nov 03 20:21:55.763115 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:55.763074 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" path="/var/lib/kubelet/pods/2690f13c-4a95-48d3-9b47-598b87d5d992/volumes" Nov 03 20:21:58.866772 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:58.866723 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" podUID="2f09c9a5-5bfe-4078-912e-48b2569e2d76" containerName="payload-processing" containerID="cri-o://967a718ba0402b60603d427fc948ef2b641f124b8a6568d94376fd0e0b632271" gracePeriod=25 Nov 03 20:21:59.325679 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.325650 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" Nov 03 20:21:59.494186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.494133 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f09c9a5-5bfe-4078-912e-48b2569e2d76-kube-api-access-w76wx\" (UniqueName: \"kubernetes.io/projected/2f09c9a5-5bfe-4078-912e-48b2569e2d76-kube-api-access-w76wx\") pod \"2f09c9a5-5bfe-4078-912e-48b2569e2d76\" (UID: \"2f09c9a5-5bfe-4078-912e-48b2569e2d76\") " Nov 03 20:21:59.494408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.494207 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f09c9a5-5bfe-4078-912e-48b2569e2d76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f09c9a5-5bfe-4078-912e-48b2569e2d76-plugins-config-volume\") pod \"2f09c9a5-5bfe-4078-912e-48b2569e2d76\" (UID: \"2f09c9a5-5bfe-4078-912e-48b2569e2d76\") " Nov 03 20:21:59.494693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.494665 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f09c9a5-5bfe-4078-912e-48b2569e2d76-plugins-config-volume" pod "2f09c9a5-5bfe-4078-912e-48b2569e2d76" (UID: "2f09c9a5-5bfe-4078-912e-48b2569e2d76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:21:59.495091 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.495067 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f09c9a5-5bfe-4078-912e-48b2569e2d76-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:59.496523 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.496496 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f09c9a5-5bfe-4078-912e-48b2569e2d76-kube-api-access-w76wx" pod "2f09c9a5-5bfe-4078-912e-48b2569e2d76" (UID: "2f09c9a5-5bfe-4078-912e-48b2569e2d76"). InnerVolumeSpecName "kube-api-access-w76wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:21:59.596767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.596704 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w76wx\" (UniqueName: \"kubernetes.io/projected/2f09c9a5-5bfe-4078-912e-48b2569e2d76-kube-api-access-w76wx\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:21:59.716115 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.716087 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" Nov 03 20:21:59.716279 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.716153 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" Nov 03 20:21:59.886931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.886832 4457 generic.go:356] "Generic (PLEG): container finished" podID="2f09c9a5-5bfe-4078-912e-48b2569e2d76" containerID="967a718ba0402b60603d427fc948ef2b641f124b8a6568d94376fd0e0b632271" exitCode=0 Nov 03 20:21:59.886931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.886878 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" Nov 03 20:21:59.887514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.887095 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" event={"ID":"2f09c9a5-5bfe-4078-912e-48b2569e2d76","Type":"ContainerDied","Data":"967a718ba0402b60603d427fc948ef2b641f124b8a6568d94376fd0e0b632271"} Nov 03 20:21:59.887514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.887136 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws" event={"ID":"2f09c9a5-5bfe-4078-912e-48b2569e2d76","Type":"ContainerDied","Data":"ec1669f8e3b97b01c4ecaeef05a52a9d71cbbf89cbfdff672967911bf28893e4"} Nov 03 20:21:59.887514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.887158 4457 scope.go:122] "RemoveContainer" containerID="967a718ba0402b60603d427fc948ef2b641f124b8a6568d94376fd0e0b632271" Nov 03 20:21:59.901260 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.901089 4457 scope.go:122] "RemoveContainer" containerID="967a718ba0402b60603d427fc948ef2b641f124b8a6568d94376fd0e0b632271" Nov 03 20:21:59.901454 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:21:59.901433 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967a718ba0402b60603d427fc948ef2b641f124b8a6568d94376fd0e0b632271\": container with ID starting with 967a718ba0402b60603d427fc948ef2b641f124b8a6568d94376fd0e0b632271 not found: ID does not exist" containerID="967a718ba0402b60603d427fc948ef2b641f124b8a6568d94376fd0e0b632271" Nov 03 20:21:59.901544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.901464 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967a718ba0402b60603d427fc948ef2b641f124b8a6568d94376fd0e0b632271"} err="failed to get container status \"967a718ba0402b60603d427fc948ef2b641f124b8a6568d94376fd0e0b632271\": rpc error: code = NotFound desc = could not find container \"967a718ba0402b60603d427fc948ef2b641f124b8a6568d94376fd0e0b632271\": container with ID starting with 967a718ba0402b60603d427fc948ef2b641f124b8a6568d94376fd0e0b632271 not found: ID does not exist" Nov 03 20:21:59.925255 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.925228 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws"] Nov 03 20:21:59.934423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.934395 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4"] Nov 03 20:21:59.935215 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.935194 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" Nov 03 20:21:59.935215 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.935215 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" Nov 03 20:21:59.935374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.935239 4457 cpu_manager.go:414] "containerMap: removing container" podUID="426daf06-f207-404a-8361-395cb6745958" containerName="revoke-keys" Nov 03 20:21:59.935374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.935245 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426daf06-f207-404a-8361-395cb6745958" containerName="revoke-keys" Nov 03 20:21:59.935374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.935281 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2f09c9a5-5bfe-4078-912e-48b2569e2d76" containerName="payload-processing" Nov 03 20:21:59.935374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.935287 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f09c9a5-5bfe-4078-912e-48b2569e2d76" containerName="payload-processing" Nov 03 20:21:59.935541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.935428 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="426daf06-f207-404a-8361-395cb6745958" containerName="revoke-keys" Nov 03 20:21:59.935541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.935448 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f09c9a5-5bfe-4078-912e-48b2569e2d76" containerName="payload-processing" Nov 03 20:21:59.935541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.935456 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="2690f13c-4a95-48d3-9b47-598b87d5d992" containerName="istio-proxy" Nov 03 20:21:59.940341 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.940324 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-06294c6a-558877d94b-v8bws"] Nov 03 20:21:59.940431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.940418 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" Nov 03 20:21:59.947810 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.947788 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-15e02e6b\"" Nov 03 20:21:59.953166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.953144 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-15e02e6b-dockercfg-6khbz\"" Nov 03 20:21:59.954874 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.954855 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4"] Nov 03 20:21:59.963419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.963382 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77"] Nov 03 20:21:59.982719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.982680 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" Nov 03 20:21:59.986182 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:21:59.986134 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77"] Nov 03 20:22:00.119410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.118642 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx2rh\" (UniqueName: \"kubernetes.io/projected/52bb2310-0cce-4dc0-97ac-484f919f099c-kube-api-access-nx2rh\") pod \"payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4\" (UID: \"52bb2310-0cce-4dc0-97ac-484f919f099c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" Nov 03 20:22:00.119410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.118749 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bb2310-0cce-4dc0-97ac-484f919f099c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4\" (UID: \"52bb2310-0cce-4dc0-97ac-484f919f099c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" Nov 03 20:22:00.119410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.118822 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-plugins-config-volume\") pod \"payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77\" (UID: \"4c9fe6d7-437e-4ef6-97dd-22e7138b8690\") " pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" Nov 03 20:22:00.119410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.118911 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qxh\" (UniqueName: \"kubernetes.io/projected/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-kube-api-access-m5qxh\") pod \"payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77\" (UID: \"4c9fe6d7-437e-4ef6-97dd-22e7138b8690\") " pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" Nov 03 20:22:00.226485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.225482 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qxh\" (UniqueName: \"kubernetes.io/projected/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-kube-api-access-m5qxh\") pod \"payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77\" (UID: \"4c9fe6d7-437e-4ef6-97dd-22e7138b8690\") " pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" Nov 03 20:22:00.226485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.225765 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx2rh\" (UniqueName: \"kubernetes.io/projected/52bb2310-0cce-4dc0-97ac-484f919f099c-kube-api-access-nx2rh\") pod \"payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4\" (UID: \"52bb2310-0cce-4dc0-97ac-484f919f099c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" Nov 03 20:22:00.226485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.225844 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bb2310-0cce-4dc0-97ac-484f919f099c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4\" (UID: \"52bb2310-0cce-4dc0-97ac-484f919f099c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" Nov 03 20:22:00.226485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.225917 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-plugins-config-volume\") pod \"payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77\" (UID: \"4c9fe6d7-437e-4ef6-97dd-22e7138b8690\") " pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" Nov 03 20:22:00.237456 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.227025 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-plugins-config-volume\") pod \"payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77\" (UID: \"4c9fe6d7-437e-4ef6-97dd-22e7138b8690\") " pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" Nov 03 20:22:00.237456 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.230346 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bb2310-0cce-4dc0-97ac-484f919f099c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4\" (UID: \"52bb2310-0cce-4dc0-97ac-484f919f099c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" Nov 03 20:22:00.255289 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.249992 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qxh\" (UniqueName: \"kubernetes.io/projected/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-kube-api-access-m5qxh\") pod \"payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77\" (UID: \"4c9fe6d7-437e-4ef6-97dd-22e7138b8690\") " pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" Nov 03 20:22:00.264405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.257061 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx2rh\" (UniqueName: \"kubernetes.io/projected/52bb2310-0cce-4dc0-97ac-484f919f099c-kube-api-access-nx2rh\") pod \"payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4\" (UID: \"52bb2310-0cce-4dc0-97ac-484f919f099c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" Nov 03 20:22:00.264405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.259714 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" Nov 03 20:22:00.306231 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.306200 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" Nov 03 20:22:00.788583 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:22:00.788555 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52bb2310_0cce_4dc0_97ac_484f919f099c.slice/crio-ed39122fd0c82b71405e8c5b541bed85c4b9729de9046ac7deece986af167f9b WatchSource:0}: Error finding container ed39122fd0c82b71405e8c5b541bed85c4b9729de9046ac7deece986af167f9b: Status 404 returned error can't find the container with id ed39122fd0c82b71405e8c5b541bed85c4b9729de9046ac7deece986af167f9b Nov 03 20:22:00.790436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.790363 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4"] Nov 03 20:22:00.818914 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.818880 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77"] Nov 03 20:22:00.821197 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:22:00.821173 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9fe6d7_437e_4ef6_97dd_22e7138b8690.slice/crio-ceb7eac8f5168503858cdf7120feed60c015b049d89f72c5f65d988acccbddd1 WatchSource:0}: Error finding container ceb7eac8f5168503858cdf7120feed60c015b049d89f72c5f65d988acccbddd1: Status 404 returned error can't find the container with id ceb7eac8f5168503858cdf7120feed60c015b049d89f72c5f65d988acccbddd1 Nov 03 20:22:00.899526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.899497 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" event={"ID":"4c9fe6d7-437e-4ef6-97dd-22e7138b8690","Type":"ContainerStarted","Data":"ceb7eac8f5168503858cdf7120feed60c015b049d89f72c5f65d988acccbddd1"} Nov 03 20:22:00.901451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.901422 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" event={"ID":"52bb2310-0cce-4dc0-97ac-484f919f099c","Type":"ContainerStarted","Data":"b4ec95391cff73ab8485d7a5a653b5dce82058d25ed12524321d8a6ddd2055a7"} Nov 03 20:22:00.901570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.901474 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" event={"ID":"52bb2310-0cce-4dc0-97ac-484f919f099c","Type":"ContainerStarted","Data":"ed39122fd0c82b71405e8c5b541bed85c4b9729de9046ac7deece986af167f9b"} Nov 03 20:22:00.901570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.901510 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" Nov 03 20:22:00.922780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:00.922721 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" podStartSLOduration=1.9227022649999999 podStartE2EDuration="1.922702265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:00.919321856 +0000 UTC m=+1499.294918604" watchObservedRunningTime="2026-11-03 20:22:00.922702265 +0000 UTC m=+1499.298299013" Nov 03 20:22:01.653533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.653500 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb"] Nov 03 20:22:01.659715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.659685 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" Nov 03 20:22:01.667886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.667854 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb"] Nov 03 20:22:01.683601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.683507 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb\" (UID: \"c8037473-08c1-4d3c-a9d6-8d61810da17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" Nov 03 20:22:01.687858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.684495 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb\" (UID: \"c8037473-08c1-4d3c-a9d6-8d61810da17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" Nov 03 20:22:01.687858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.684727 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qj62\" (UniqueName: \"kubernetes.io/projected/c8037473-08c1-4d3c-a9d6-8d61810da17f-kube-api-access-8qj62\") pod \"maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb\" (UID: \"c8037473-08c1-4d3c-a9d6-8d61810da17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" Nov 03 20:22:01.742582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.742540 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:22:01.742964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.742946 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:22:01.786037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.785998 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb\" (UID: \"c8037473-08c1-4d3c-a9d6-8d61810da17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" Nov 03 20:22:01.786236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.786108 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb\" (UID: \"c8037473-08c1-4d3c-a9d6-8d61810da17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" Nov 03 20:22:01.786477 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.786451 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qj62\" (UniqueName: \"kubernetes.io/projected/c8037473-08c1-4d3c-a9d6-8d61810da17f-kube-api-access-8qj62\") pod \"maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb\" (UID: \"c8037473-08c1-4d3c-a9d6-8d61810da17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" Nov 03 20:22:01.787189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.787163 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb\" (UID: \"c8037473-08c1-4d3c-a9d6-8d61810da17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" Nov 03 20:22:01.824832 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.824796 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f09c9a5-5bfe-4078-912e-48b2569e2d76" path="/var/lib/kubelet/pods/2f09c9a5-5bfe-4078-912e-48b2569e2d76/volumes" Nov 03 20:22:01.828030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.828005 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb\" (UID: \"c8037473-08c1-4d3c-a9d6-8d61810da17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" Nov 03 20:22:01.832093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.832063 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qj62\" (UniqueName: \"kubernetes.io/projected/c8037473-08c1-4d3c-a9d6-8d61810da17f-kube-api-access-8qj62\") pod \"maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb\" (UID: \"c8037473-08c1-4d3c-a9d6-8d61810da17f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" Nov 03 20:22:01.914421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.914310 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" event={"ID":"4c9fe6d7-437e-4ef6-97dd-22e7138b8690","Type":"ContainerStarted","Data":"ab3948abd4ace9e506bc5ba0c4e3ee9e1d52080449ba7530e6cc85bbfe69d11b"} Nov 03 20:22:01.914982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.914942 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" Nov 03 20:22:01.941056 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.940989 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" podStartSLOduration=2.940970197 podStartE2EDuration="2.940970197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:21:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:01.933765508 +0000 UTC m=+1500.309362280" watchObservedRunningTime="2026-11-03 20:22:01.940970197 +0000 UTC m=+1500.316566948" Nov 03 20:22:01.992271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:01.992238 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" Nov 03 20:22:02.465067 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:02.463936 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j"] Nov 03 20:22:02.465067 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:02.464203 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" containerID="cri-o://290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38" gracePeriod=30 Nov 03 20:22:02.530256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:02.530182 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:22:02.536897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:02.536872 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:22:02.553830 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:02.553806 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:22:02.558488 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:02.558464 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:22:02.566578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:02.566530 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb"] Nov 03 20:22:02.572113 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:22:02.572089 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8037473_08c1_4d3c_a9d6_8d61810da17f.slice/crio-48d9c8da37e64f7b814d5be571981fbb38d91c458cd193d77ff3002956aecc6b WatchSource:0}: Error finding container 48d9c8da37e64f7b814d5be571981fbb38d91c458cd193d77ff3002956aecc6b: Status 404 returned error can't find the container with id 48d9c8da37e64f7b814d5be571981fbb38d91c458cd193d77ff3002956aecc6b Nov 03 20:22:02.935412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:02.935290 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" event={"ID":"c8037473-08c1-4d3c-a9d6-8d61810da17f","Type":"ContainerStarted","Data":"48d9c8da37e64f7b814d5be571981fbb38d91c458cd193d77ff3002956aecc6b"} Nov 03 20:22:03.950723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:03.950686 4457 generic.go:356] "Generic (PLEG): container finished" podID="c8037473-08c1-4d3c-a9d6-8d61810da17f" containerID="097d6d84be7b51d32f9b41a9ddcfb798c44838b8478310ad9ac68bcebb554cc8" exitCode=0 Nov 03 20:22:03.951149 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:03.950793 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" event={"ID":"c8037473-08c1-4d3c-a9d6-8d61810da17f","Type":"ContainerDied","Data":"097d6d84be7b51d32f9b41a9ddcfb798c44838b8478310ad9ac68bcebb554cc8"} Nov 03 20:22:05.315697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.313597 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4"] Nov 03 20:22:05.330995 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.321539 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06294c6a-cqkf4"] Nov 03 20:22:05.556832 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.556803 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" Nov 03 20:22:05.659970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.659799 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-openshift-ca\") pod \"c8037473-08c1-4d3c-a9d6-8d61810da17f\" (UID: \"c8037473-08c1-4d3c-a9d6-8d61810da17f\") " Nov 03 20:22:05.659970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.659851 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8037473-08c1-4d3c-a9d6-8d61810da17f-kube-api-access-8qj62\" (UniqueName: \"kubernetes.io/projected/c8037473-08c1-4d3c-a9d6-8d61810da17f-kube-api-access-8qj62\") pod \"c8037473-08c1-4d3c-a9d6-8d61810da17f\" (UID: \"c8037473-08c1-4d3c-a9d6-8d61810da17f\") " Nov 03 20:22:05.660205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.659973 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-xks-ca\") pod \"c8037473-08c1-4d3c-a9d6-8d61810da17f\" (UID: \"c8037473-08c1-4d3c-a9d6-8d61810da17f\") " Nov 03 20:22:05.660272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.660231 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-openshift-ca" pod "c8037473-08c1-4d3c-a9d6-8d61810da17f" (UID: "c8037473-08c1-4d3c-a9d6-8d61810da17f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:05.660680 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.660654 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:05.662285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.662258 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8037473-08c1-4d3c-a9d6-8d61810da17f-kube-api-access-8qj62" pod "c8037473-08c1-4d3c-a9d6-8d61810da17f" (UID: "c8037473-08c1-4d3c-a9d6-8d61810da17f"). InnerVolumeSpecName "kube-api-access-8qj62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:05.662285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.662273 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-xks-ca" pod "c8037473-08c1-4d3c-a9d6-8d61810da17f" (UID: "c8037473-08c1-4d3c-a9d6-8d61810da17f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:05.762203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.762159 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qj62\" (UniqueName: \"kubernetes.io/projected/c8037473-08c1-4d3c-a9d6-8d61810da17f-kube-api-access-8qj62\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:05.762203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.762196 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8037473-08c1-4d3c-a9d6-8d61810da17f-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:05.765674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.765639 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426daf06-f207-404a-8361-395cb6745958" path="/var/lib/kubelet/pods/426daf06-f207-404a-8361-395cb6745958/volumes" Nov 03 20:22:05.984268 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.984174 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" Nov 03 20:22:05.984268 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.984199 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb" event={"ID":"c8037473-08c1-4d3c-a9d6-8d61810da17f","Type":"ContainerDied","Data":"48d9c8da37e64f7b814d5be571981fbb38d91c458cd193d77ff3002956aecc6b"} Nov 03 20:22:05.984268 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:05.984259 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48d9c8da37e64f7b814d5be571981fbb38d91c458cd193d77ff3002956aecc6b" Nov 03 20:22:06.148624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:06.148591 4457 scope.go:122] "RemoveContainer" containerID="db08e05e6a2b8a5ac991f850c3d46f42c65c994e252f4f0a483effd60f85b9b9" Nov 03 20:22:06.157068 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:22:06.157032 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:22:06.162041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:06.162017 4457 scope.go:122] "RemoveContainer" containerID="ecdd6639fd6fc016518fba2697ae56319e1b622987a8d0640b0f8777b0cda741" Nov 03 20:22:06.177051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:06.177025 4457 scope.go:122] "RemoveContainer" containerID="b94d1811087b324df1941ac2fe882ecd924afc5f15254a7d7f3b114e2cee7237" Nov 03 20:22:06.190757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:06.190730 4457 scope.go:122] "RemoveContainer" containerID="98dea328c5ec30894c9ab86c9e326323c159a22ceaa322b17248970afafc8441" Nov 03 20:22:06.202031 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:06.202006 4457 scope.go:122] "RemoveContainer" containerID="e9486c42f7d40554102c62042b9a826268c5fe9d9df0142f2e6c9d12e0292616" Nov 03 20:22:06.214858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:06.214834 4457 scope.go:122] "RemoveContainer" containerID="496cf619788d0b0d33498a12d17650336d944e276550e3585b458e2964d26e35" Nov 03 20:22:06.241004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:06.240974 4457 scope.go:122] "RemoveContainer" containerID="c96b763264634f7e7a269d1dca08c2f68ffc87340f16817e94191ee761b55619" Nov 03 20:22:06.253559 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:06.253534 4457 scope.go:122] "RemoveContainer" containerID="479d2d78019fc8eeca15bd6f4d4eb0b547fe8e9191a61a3bda3fedb56b956aa6" Nov 03 20:22:06.267618 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:06.267593 4457 scope.go:122] "RemoveContainer" containerID="5137cd74fe509c2c61bcf508ffd51e64d6081b0e716cb3cee6793f5c383ccca2" Nov 03 20:22:06.280664 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:06.280637 4457 scope.go:122] "RemoveContainer" containerID="dd649d122aee7dac604de6ffe3bef55fde84796158ae46964aa811fc0d2fc078" Nov 03 20:22:07.599099 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:22:07.599061 4457 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af92d22_ef86_41f9_a91e_2f5b6653bad6.slice/crio-conmon-290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:22:07.783208 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.783180 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:22:07.815924 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.815887 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istiod-ca-cert\") pod \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " Nov 03 20:22:07.816102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.816075 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-credential-socket\") pod \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " Nov 03 20:22:07.816173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.816114 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-data\") pod \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " Nov 03 20:22:07.816173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.816165 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-kube-api-access-r6l2d\" (UniqueName: \"kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-kube-api-access-r6l2d\") pod \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " Nov 03 20:22:07.816262 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.816239 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-envoy\") pod \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " Nov 03 20:22:07.816314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.816281 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-podinfo\") pod \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " Nov 03 20:22:07.816375 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.816319 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istiod-ca-cert" pod "0af92d22-ef86-41f9-a91e-2f5b6653bad6" (UID: "0af92d22-ef86-41f9-a91e-2f5b6653bad6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:07.816562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.816537 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-data" pod "0af92d22-ef86-41f9-a91e-2f5b6653bad6" (UID: "0af92d22-ef86-41f9-a91e-2f5b6653bad6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:07.816619 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.816584 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-token\" (UniqueName: \"kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-token\") pod \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " Nov 03 20:22:07.816676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.816630 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-certs\") pod \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " Nov 03 20:22:07.816759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.816740 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-socket\") pod \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\" (UID: \"0af92d22-ef86-41f9-a91e-2f5b6653bad6\") " Nov 03 20:22:07.816830 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.816807 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-credential-socket" pod "0af92d22-ef86-41f9-a91e-2f5b6653bad6" (UID: "0af92d22-ef86-41f9-a91e-2f5b6653bad6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:07.817061 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.817036 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-socket" pod "0af92d22-ef86-41f9-a91e-2f5b6653bad6" (UID: "0af92d22-ef86-41f9-a91e-2f5b6653bad6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:07.818267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.817959 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.818267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.818074 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-certs" pod "0af92d22-ef86-41f9-a91e-2f5b6653bad6" (UID: "0af92d22-ef86-41f9-a91e-2f5b6653bad6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:07.818267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.818094 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.818267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.818118 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.818267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.818133 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.819013 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.818989 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-podinfo" pod "0af92d22-ef86-41f9-a91e-2f5b6653bad6" (UID: "0af92d22-ef86-41f9-a91e-2f5b6653bad6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:22:07.819798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.819768 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-envoy" pod "0af92d22-ef86-41f9-a91e-2f5b6653bad6" (UID: "0af92d22-ef86-41f9-a91e-2f5b6653bad6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:07.820205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.820181 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-kube-api-access-r6l2d" pod "0af92d22-ef86-41f9-a91e-2f5b6653bad6" (UID: "0af92d22-ef86-41f9-a91e-2f5b6653bad6"). InnerVolumeSpecName "kube-api-access-r6l2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:07.820432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.820407 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-token" pod "0af92d22-ef86-41f9-a91e-2f5b6653bad6" (UID: "0af92d22-ef86-41f9-a91e-2f5b6653bad6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:07.921850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.921708 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6l2d\" (UniqueName: \"kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-kube-api-access-r6l2d\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.921850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.921747 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.921850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.921760 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.922588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.922429 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0af92d22-ef86-41f9-a91e-2f5b6653bad6-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:07.922689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:07.922468 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0af92d22-ef86-41f9-a91e-2f5b6653bad6-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:08.010840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:08.010751 4457 generic.go:356] "Generic (PLEG): container finished" podID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerID="290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38" exitCode=0 Nov 03 20:22:08.010840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:08.010810 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" Nov 03 20:22:08.011011 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:08.010853 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" event={"ID":"0af92d22-ef86-41f9-a91e-2f5b6653bad6","Type":"ContainerDied","Data":"290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38"} Nov 03 20:22:08.011011 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:08.010925 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j" event={"ID":"0af92d22-ef86-41f9-a91e-2f5b6653bad6","Type":"ContainerDied","Data":"68f9c7749b164a15c33051f2fab5def386dbf112c2eaa21808fa57f375999fe5"} Nov 03 20:22:08.011011 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:08.010957 4457 scope.go:122] "RemoveContainer" containerID="290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38" Nov 03 20:22:08.023989 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:08.023968 4457 scope.go:122] "RemoveContainer" containerID="290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38" Nov 03 20:22:08.024355 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:22:08.024293 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38\": container with ID starting with 290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38 not found: ID does not exist" containerID="290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38" Nov 03 20:22:08.024355 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:08.024329 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38"} err="failed to get container status \"290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38\": rpc error: code = NotFound desc = could not find container \"290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38\": container with ID starting with 290f1e1d542e421c31d1b9dcec26edb9fe64f11404ea98e638b59d27decd2c38 not found: ID does not exist" Nov 03 20:22:08.042675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:08.042578 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j"] Nov 03 20:22:08.051601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:08.051577 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-wqp9j"] Nov 03 20:22:09.772675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:09.772641 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" path="/var/lib/kubelet/pods/0af92d22-ef86-41f9-a91e-2f5b6653bad6/volumes" Nov 03 20:22:11.781420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:11.781373 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25"] Nov 03 20:22:11.829588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:11.829551 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw"] Nov 03 20:22:11.917539 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:11.917512 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" Nov 03 20:22:12.066508 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:12.066462 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" podUID="3a580169-1ed3-4194-9bee-19a153b22984" containerName="istio-proxy" containerID="cri-o://1b452f46f109d18918c1ab574c8a41d34a67fbf058244d32eab1ad1efa12fd97" gracePeriod=30 Nov 03 20:22:12.939454 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:12.939425 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" Nov 03 20:22:16.830289 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:16.830220 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" podUID="baa9ebd7-1105-40cc-b835-344cf70328e0" containerName="payload-processing" containerID="cri-o://ee764969818185f2f08a3dd20b47da573dd2a2999b1cb0ea993c157386a1aa8e" gracePeriod=25 Nov 03 20:22:17.215460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.215430 4457 generic.go:356] "Generic (PLEG): container finished" podID="baa9ebd7-1105-40cc-b835-344cf70328e0" containerID="ee764969818185f2f08a3dd20b47da573dd2a2999b1cb0ea993c157386a1aa8e" exitCode=0 Nov 03 20:22:17.215597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.215517 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" event={"ID":"baa9ebd7-1105-40cc-b835-344cf70328e0","Type":"ContainerDied","Data":"ee764969818185f2f08a3dd20b47da573dd2a2999b1cb0ea993c157386a1aa8e"} Nov 03 20:22:17.217462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.217374 4457 generic.go:356] "Generic (PLEG): container finished" podID="3a580169-1ed3-4194-9bee-19a153b22984" containerID="1b452f46f109d18918c1ab574c8a41d34a67fbf058244d32eab1ad1efa12fd97" exitCode=0 Nov 03 20:22:17.217658 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.217642 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" event={"ID":"3a580169-1ed3-4194-9bee-19a153b22984","Type":"ContainerDied","Data":"1b452f46f109d18918c1ab574c8a41d34a67fbf058244d32eab1ad1efa12fd97"} Nov 03 20:22:17.244513 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.244483 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" Nov 03 20:22:17.386467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.374186 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa9ebd7-1105-40cc-b835-344cf70328e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa9ebd7-1105-40cc-b835-344cf70328e0-plugins-config-volume\") pod \"baa9ebd7-1105-40cc-b835-344cf70328e0\" (UID: \"baa9ebd7-1105-40cc-b835-344cf70328e0\") " Nov 03 20:22:17.386467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.374297 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa9ebd7-1105-40cc-b835-344cf70328e0-kube-api-access-t74qr\" (UniqueName: \"kubernetes.io/projected/baa9ebd7-1105-40cc-b835-344cf70328e0-kube-api-access-t74qr\") pod \"baa9ebd7-1105-40cc-b835-344cf70328e0\" (UID: \"baa9ebd7-1105-40cc-b835-344cf70328e0\") " Nov 03 20:22:17.402413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.387575 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa9ebd7-1105-40cc-b835-344cf70328e0-kube-api-access-t74qr" pod "baa9ebd7-1105-40cc-b835-344cf70328e0" (UID: "baa9ebd7-1105-40cc-b835-344cf70328e0"). InnerVolumeSpecName "kube-api-access-t74qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:17.402413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.387991 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa9ebd7-1105-40cc-b835-344cf70328e0-plugins-config-volume" pod "baa9ebd7-1105-40cc-b835-344cf70328e0" (UID: "baa9ebd7-1105-40cc-b835-344cf70328e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:17.479374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.476517 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baa9ebd7-1105-40cc-b835-344cf70328e0-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:17.479374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.476540 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t74qr\" (UniqueName: \"kubernetes.io/projected/baa9ebd7-1105-40cc-b835-344cf70328e0-kube-api-access-t74qr\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:17.486451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.485890 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:22:17.577696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.577656 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-envoy\") pod \"3a580169-1ed3-4194-9bee-19a153b22984\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " Nov 03 20:22:17.577902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.577722 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-credential-socket\") pod \"3a580169-1ed3-4194-9bee-19a153b22984\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " Nov 03 20:22:17.577902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.577763 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a580169-1ed3-4194-9bee-19a153b22984-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a580169-1ed3-4194-9bee-19a153b22984-istiod-ca-cert\") pod \"3a580169-1ed3-4194-9bee-19a153b22984\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " Nov 03 20:22:17.577902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.577860 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-socket\") pod \"3a580169-1ed3-4194-9bee-19a153b22984\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " Nov 03 20:22:17.578106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.577915 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-certs\") pod \"3a580169-1ed3-4194-9bee-19a153b22984\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " Nov 03 20:22:17.578106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.577959 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a580169-1ed3-4194-9bee-19a153b22984-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a580169-1ed3-4194-9bee-19a153b22984-istio-podinfo\") pod \"3a580169-1ed3-4194-9bee-19a153b22984\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " Nov 03 20:22:17.578106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.578052 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-istio-token\" (UniqueName: \"kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-istio-token\") pod \"3a580169-1ed3-4194-9bee-19a153b22984\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " Nov 03 20:22:17.578264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.578138 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-kube-api-access-btlcq\" (UniqueName: \"kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-kube-api-access-btlcq\") pod \"3a580169-1ed3-4194-9bee-19a153b22984\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " Nov 03 20:22:17.578264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.578170 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-data\") pod \"3a580169-1ed3-4194-9bee-19a153b22984\" (UID: \"3a580169-1ed3-4194-9bee-19a153b22984\") " Nov 03 20:22:17.579218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.579187 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-data" pod "3a580169-1ed3-4194-9bee-19a153b22984" (UID: "3a580169-1ed3-4194-9bee-19a153b22984"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:17.579852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.579825 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-certs" pod "3a580169-1ed3-4194-9bee-19a153b22984" (UID: "3a580169-1ed3-4194-9bee-19a153b22984"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:17.580041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.580021 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-credential-socket" pod "3a580169-1ed3-4194-9bee-19a153b22984" (UID: "3a580169-1ed3-4194-9bee-19a153b22984"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:17.580344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.580325 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a580169-1ed3-4194-9bee-19a153b22984-istiod-ca-cert" pod "3a580169-1ed3-4194-9bee-19a153b22984" (UID: "3a580169-1ed3-4194-9bee-19a153b22984"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:17.580585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.580564 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-socket" pod "3a580169-1ed3-4194-9bee-19a153b22984" (UID: "3a580169-1ed3-4194-9bee-19a153b22984"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:17.582958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.582930 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-envoy" pod "3a580169-1ed3-4194-9bee-19a153b22984" (UID: "3a580169-1ed3-4194-9bee-19a153b22984"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:22:17.583204 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.583182 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-istio-token" pod "3a580169-1ed3-4194-9bee-19a153b22984" (UID: "3a580169-1ed3-4194-9bee-19a153b22984"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:17.584168 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.584141 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a580169-1ed3-4194-9bee-19a153b22984-istio-podinfo" pod "3a580169-1ed3-4194-9bee-19a153b22984" (UID: "3a580169-1ed3-4194-9bee-19a153b22984"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:22:17.584823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.584798 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-kube-api-access-btlcq" pod "3a580169-1ed3-4194-9bee-19a153b22984" (UID: "3a580169-1ed3-4194-9bee-19a153b22984"). InnerVolumeSpecName "kube-api-access-btlcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:17.681026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.680935 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:17.681026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.680980 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:17.681026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.680994 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a580169-1ed3-4194-9bee-19a153b22984-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:17.681026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.681009 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:17.681026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.681024 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btlcq\" (UniqueName: \"kubernetes.io/projected/3a580169-1ed3-4194-9bee-19a153b22984-kube-api-access-btlcq\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:17.681307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.681038 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:17.681307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.681052 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:17.681307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.681066 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a580169-1ed3-4194-9bee-19a153b22984-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:17.681307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:17.681081 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a580169-1ed3-4194-9bee-19a153b22984-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:18.030864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.030786 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b"] Nov 03 20:22:18.031578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.031561 4457 cpu_manager.go:414] "containerMap: removing container" podUID="3a580169-1ed3-4194-9bee-19a153b22984" containerName="istio-proxy" Nov 03 20:22:18.031578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.031579 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a580169-1ed3-4194-9bee-19a153b22984" containerName="istio-proxy" Nov 03 20:22:18.031687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.031609 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c8037473-08c1-4d3c-a9d6-8d61810da17f" containerName="revoke-keys" Nov 03 20:22:18.031687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.031614 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8037473-08c1-4d3c-a9d6-8d61810da17f" containerName="revoke-keys" Nov 03 20:22:18.031687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.031626 4457 cpu_manager.go:414] "containerMap: removing container" podUID="baa9ebd7-1105-40cc-b835-344cf70328e0" containerName="payload-processing" Nov 03 20:22:18.031687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.031632 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa9ebd7-1105-40cc-b835-344cf70328e0" containerName="payload-processing" Nov 03 20:22:18.031687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.031658 4457 cpu_manager.go:414] "containerMap: removing container" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" Nov 03 20:22:18.031687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.031664 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" Nov 03 20:22:18.031875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.031774 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a580169-1ed3-4194-9bee-19a153b22984" containerName="istio-proxy" Nov 03 20:22:18.031875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.031788 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8037473-08c1-4d3c-a9d6-8d61810da17f" containerName="revoke-keys" Nov 03 20:22:18.031875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.031795 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa9ebd7-1105-40cc-b835-344cf70328e0" containerName="payload-processing" Nov 03 20:22:18.031875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.031804 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af92d22-ef86-41f9-a91e-2f5b6653bad6" containerName="istio-proxy" Nov 03 20:22:18.036844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.036819 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.044536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.044510 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-15e02e6b-openshift-default-dockercfg-mjzbc\"" Nov 03 20:22:18.049150 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.049123 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b"] Nov 03 20:22:18.193351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.193303 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.193600 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.193382 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.193600 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.193499 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.193739 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.193689 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.193796 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.193736 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qlzm\" (UniqueName: \"kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-kube-api-access-5qlzm\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.193850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.193806 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.193917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.193874 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.193997 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.193970 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.194063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.194017 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.227936 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.227909 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" Nov 03 20:22:18.227936 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.227915 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw" event={"ID":"baa9ebd7-1105-40cc-b835-344cf70328e0","Type":"ContainerDied","Data":"57fa1877ada671f46d742035a56f8f26bcdba7939804de6f8e54f364b4e3dd12"} Nov 03 20:22:18.228160 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.227981 4457 scope.go:122] "RemoveContainer" containerID="ee764969818185f2f08a3dd20b47da573dd2a2999b1cb0ea993c157386a1aa8e" Nov 03 20:22:18.229888 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.229844 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" Nov 03 20:22:18.229888 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.229855 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25" event={"ID":"3a580169-1ed3-4194-9bee-19a153b22984","Type":"ContainerDied","Data":"ce3fded768478ac43138a40c1eb3c8b4513bcad205324e4eb9e08ccbc85a2e1d"} Nov 03 20:22:18.240948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.240930 4457 scope.go:122] "RemoveContainer" containerID="1b452f46f109d18918c1ab574c8a41d34a67fbf058244d32eab1ad1efa12fd97" Nov 03 20:22:18.259127 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.259106 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25"] Nov 03 20:22:18.300464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.294148 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-npr25"] Nov 03 20:22:18.300464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.295311 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.300464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.295398 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.300464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.295456 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.300464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.295613 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.300464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.295648 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qlzm\" (UniqueName: \"kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-kube-api-access-5qlzm\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.300464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.295707 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.300464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.295917 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.300464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.296032 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.300464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.296075 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.300464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.297023 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.306410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.304462 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.306410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.304750 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.320421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.306669 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.320421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.307138 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.320421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.309404 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw"] Nov 03 20:22:18.330408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.326204 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.338457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.330723 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d5927-85cdddc98d-fs5cw"] Nov 03 20:22:18.338457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.331796 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.338457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.331989 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.338457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.335492 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qlzm\" (UniqueName: \"kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-kube-api-access-5qlzm\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.358623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.358592 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:18.654534 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:22:18.654499 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f5ce6fb_4dca_4cd7_befa_82ea92a023b1.slice/crio-aac3f6f54b0f7f7354a60da70f6cfd927bc2dc27506d53c89531c31154c42a00 WatchSource:0}: Error finding container aac3f6f54b0f7f7354a60da70f6cfd927bc2dc27506d53c89531c31154c42a00: Status 404 returned error can't find the container with id aac3f6f54b0f7f7354a60da70f6cfd927bc2dc27506d53c89531c31154c42a00 Nov 03 20:22:18.655226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.655198 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b"] Nov 03 20:22:18.658068 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.658030 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:18.658165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.658111 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:18.658165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:18.658159 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:22:19.242020 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.241980 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" event={"ID":"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1","Type":"ContainerStarted","Data":"4b71cf7c808ac46bbd87a5430dc33ab701f36c8fec9fe99c87c52307fc3eddb7"} Nov 03 20:22:19.242527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.242043 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" event={"ID":"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1","Type":"ContainerStarted","Data":"aac3f6f54b0f7f7354a60da70f6cfd927bc2dc27506d53c89531c31154c42a00"} Nov 03 20:22:19.267110 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.267030 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" podStartSLOduration=1.267011193 podStartE2EDuration="1.267011193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:19.265934649 +0000 UTC m=+1517.641531395" watchObservedRunningTime="2026-11-03 20:22:19.267011193 +0000 UTC m=+1517.642607940" Nov 03 20:22:19.364291 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.362933 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:19.373406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.369524 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:22:19.373406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.369588 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" podUID="1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 20:22:19.815247 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.815214 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a580169-1ed3-4194-9bee-19a153b22984" path="/var/lib/kubelet/pods/3a580169-1ed3-4194-9bee-19a153b22984/volumes" Nov 03 20:22:19.816174 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.816135 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa9ebd7-1105-40cc-b835-344cf70328e0" path="/var/lib/kubelet/pods/baa9ebd7-1105-40cc-b835-344cf70328e0/volumes" Nov 03 20:22:19.816848 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.816819 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc"] Nov 03 20:22:19.832037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.832013 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x"] Nov 03 20:22:19.834685 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.832424 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" Nov 03 20:22:19.839517 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.839496 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc"] Nov 03 20:22:19.839517 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.839508 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3ebde5\"" Nov 03 20:22:19.839731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.839531 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x"] Nov 03 20:22:19.839731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.839594 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" Nov 03 20:22:19.842886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:19.842864 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3ebde5-dockercfg-v7gp4\"" Nov 03 20:22:20.013416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.003143 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16ca003-2cef-4af2-ba76-9a3f76629867-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x\" (UID: \"f16ca003-2cef-4af2-ba76-9a3f76629867\") " pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" Nov 03 20:22:20.013416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.003206 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc\" (UID: \"69e6d47e-f7f8-4eda-8f02-8ddcb150300c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" Nov 03 20:22:20.013416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.003433 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns22b\" (UniqueName: \"kubernetes.io/projected/f16ca003-2cef-4af2-ba76-9a3f76629867-kube-api-access-ns22b\") pod \"payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x\" (UID: \"f16ca003-2cef-4af2-ba76-9a3f76629867\") " pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" Nov 03 20:22:20.013416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.003523 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhx9h\" (UniqueName: \"kubernetes.io/projected/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-kube-api-access-jhx9h\") pod \"payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc\" (UID: \"69e6d47e-f7f8-4eda-8f02-8ddcb150300c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" Nov 03 20:22:20.180463 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.179226 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns22b\" (UniqueName: \"kubernetes.io/projected/f16ca003-2cef-4af2-ba76-9a3f76629867-kube-api-access-ns22b\") pod \"payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x\" (UID: \"f16ca003-2cef-4af2-ba76-9a3f76629867\") " pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" Nov 03 20:22:20.180463 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.179304 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhx9h\" (UniqueName: \"kubernetes.io/projected/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-kube-api-access-jhx9h\") pod \"payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc\" (UID: \"69e6d47e-f7f8-4eda-8f02-8ddcb150300c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" Nov 03 20:22:20.180463 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.179495 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16ca003-2cef-4af2-ba76-9a3f76629867-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x\" (UID: \"f16ca003-2cef-4af2-ba76-9a3f76629867\") " pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" Nov 03 20:22:20.180463 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.179525 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc\" (UID: \"69e6d47e-f7f8-4eda-8f02-8ddcb150300c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" Nov 03 20:22:20.180822 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.180508 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc\" (UID: \"69e6d47e-f7f8-4eda-8f02-8ddcb150300c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" Nov 03 20:22:20.192421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.191446 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16ca003-2cef-4af2-ba76-9a3f76629867-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x\" (UID: \"f16ca003-2cef-4af2-ba76-9a3f76629867\") " pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" Nov 03 20:22:20.254604 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.253052 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhx9h\" (UniqueName: \"kubernetes.io/projected/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-kube-api-access-jhx9h\") pod \"payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc\" (UID: \"69e6d47e-f7f8-4eda-8f02-8ddcb150300c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" Nov 03 20:22:20.286546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.277076 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns22b\" (UniqueName: \"kubernetes.io/projected/f16ca003-2cef-4af2-ba76-9a3f76629867-kube-api-access-ns22b\") pod \"payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x\" (UID: \"f16ca003-2cef-4af2-ba76-9a3f76629867\") " pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" Nov 03 20:22:20.366415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.360777 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:22:20.366415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.360849 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" podUID="1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 20:22:20.402264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.399130 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" Nov 03 20:22:20.461941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.459856 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" Nov 03 20:22:20.949878 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:22:20.949835 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16ca003_2cef_4af2_ba76_9a3f76629867.slice/crio-2e31c8f82fb0d464bf01c7dd555097900fc1edacd6a3eae392b7cb9c6040a060 WatchSource:0}: Error finding container 2e31c8f82fb0d464bf01c7dd555097900fc1edacd6a3eae392b7cb9c6040a060: Status 404 returned error can't find the container with id 2e31c8f82fb0d464bf01c7dd555097900fc1edacd6a3eae392b7cb9c6040a060 Nov 03 20:22:20.951450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.951424 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x"] Nov 03 20:22:20.953521 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:22:20.953493 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e6d47e_f7f8_4eda_8f02_8ddcb150300c.slice/crio-601e52b01032263fe159ac71db4b8a0da29e95560847bf7629646481fbe7fc1d WatchSource:0}: Error finding container 601e52b01032263fe159ac71db4b8a0da29e95560847bf7629646481fbe7fc1d: Status 404 returned error can't find the container with id 601e52b01032263fe159ac71db4b8a0da29e95560847bf7629646481fbe7fc1d Nov 03 20:22:20.993051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:20.991773 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc"] Nov 03 20:22:21.364412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:21.362719 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:22:21.364412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:21.362785 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" podUID="1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 20:22:21.414411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:21.413508 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" event={"ID":"f16ca003-2cef-4af2-ba76-9a3f76629867","Type":"ContainerStarted","Data":"446bb0972bbdb2454dc22e883b03e022dc7a78b2a1ce99b5f4a54b364cf81a6b"} Nov 03 20:22:21.414411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:21.413578 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" event={"ID":"f16ca003-2cef-4af2-ba76-9a3f76629867","Type":"ContainerStarted","Data":"2e31c8f82fb0d464bf01c7dd555097900fc1edacd6a3eae392b7cb9c6040a060"} Nov 03 20:22:21.418408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:21.414840 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" Nov 03 20:22:21.457412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:21.456528 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" event={"ID":"69e6d47e-f7f8-4eda-8f02-8ddcb150300c","Type":"ContainerStarted","Data":"198c1240afc97822fc2b6d96f265ebe0d3562bee45119e94d90c582df46a3945"} Nov 03 20:22:21.457412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:21.456593 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" Nov 03 20:22:21.457412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:21.456619 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" event={"ID":"69e6d47e-f7f8-4eda-8f02-8ddcb150300c","Type":"ContainerStarted","Data":"601e52b01032263fe159ac71db4b8a0da29e95560847bf7629646481fbe7fc1d"} Nov 03 20:22:21.502411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:21.500815 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" podStartSLOduration=2.5007920930000003 podStartE2EDuration="2.500792093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:21.477809015 +0000 UTC m=+1519.853405765" watchObservedRunningTime="2026-11-03 20:22:21.500792093 +0000 UTC m=+1519.876388841" Nov 03 20:22:21.558033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:21.557306 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" podStartSLOduration=2.557288322 podStartE2EDuration="2.557288322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:21.520675763 +0000 UTC m=+1519.896272510" watchObservedRunningTime="2026-11-03 20:22:21.557288322 +0000 UTC m=+1519.932885069" Nov 03 20:22:22.363411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:22.361781 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:22:22.363411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:22.361845 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" podUID="1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 20:22:23.369409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:23.365491 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:22:23.369409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:23.365562 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" podUID="1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 20:22:24.367409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:24.366064 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:22:24.367409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:24.366281 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" podUID="1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 20:22:25.368444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:25.364361 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Nov 03 20:22:25.368444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:25.364435 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" podUID="1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Nov 03 20:22:26.654410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:26.651736 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:26.654410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:26.653566 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:26.655630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:26.655605 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:22:30.198153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.198117 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx"] Nov 03 20:22:30.204528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.204501 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" Nov 03 20:22:30.241218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.241188 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-77e83b\"" Nov 03 20:22:30.241541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.241423 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-77e83b-dockercfg-gh69l\"" Nov 03 20:22:30.246848 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.246818 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx"] Nov 03 20:22:30.262685 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.262639 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng49b\" (UniqueName: \"kubernetes.io/projected/9e5dbdf1-f78c-4368-aabb-232578d293a7-kube-api-access-ng49b\") pod \"payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx\" (UID: \"9e5dbdf1-f78c-4368-aabb-232578d293a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" Nov 03 20:22:30.262849 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.262718 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5dbdf1-f78c-4368-aabb-232578d293a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx\" (UID: \"9e5dbdf1-f78c-4368-aabb-232578d293a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" Nov 03 20:22:30.329763 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.329703 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl"] Nov 03 20:22:30.341658 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.341624 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" Nov 03 20:22:30.384719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.384634 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng49b\" (UniqueName: \"kubernetes.io/projected/9e5dbdf1-f78c-4368-aabb-232578d293a7-kube-api-access-ng49b\") pod \"payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx\" (UID: \"9e5dbdf1-f78c-4368-aabb-232578d293a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" Nov 03 20:22:30.384873 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.384728 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5dbdf1-f78c-4368-aabb-232578d293a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx\" (UID: \"9e5dbdf1-f78c-4368-aabb-232578d293a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" Nov 03 20:22:30.384873 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.384791 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5kmw\" (UniqueName: \"kubernetes.io/projected/130c5e5b-43ee-4329-ab13-f37b6c3981fb-kube-api-access-c5kmw\") pod \"payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl\" (UID: \"130c5e5b-43ee-4329-ab13-f37b6c3981fb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" Nov 03 20:22:30.384873 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.384868 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130c5e5b-43ee-4329-ab13-f37b6c3981fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl\" (UID: \"130c5e5b-43ee-4329-ab13-f37b6c3981fb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" Nov 03 20:22:30.386968 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.386878 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5dbdf1-f78c-4368-aabb-232578d293a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx\" (UID: \"9e5dbdf1-f78c-4368-aabb-232578d293a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" Nov 03 20:22:30.486851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.486746 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5kmw\" (UniqueName: \"kubernetes.io/projected/130c5e5b-43ee-4329-ab13-f37b6c3981fb-kube-api-access-c5kmw\") pod \"payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl\" (UID: \"130c5e5b-43ee-4329-ab13-f37b6c3981fb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" Nov 03 20:22:30.487144 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.487122 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130c5e5b-43ee-4329-ab13-f37b6c3981fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl\" (UID: \"130c5e5b-43ee-4329-ab13-f37b6c3981fb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" Nov 03 20:22:30.488613 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.488534 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130c5e5b-43ee-4329-ab13-f37b6c3981fb-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl\" (UID: \"130c5e5b-43ee-4329-ab13-f37b6c3981fb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" Nov 03 20:22:30.699813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.699777 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl"] Nov 03 20:22:30.728411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.728192 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5kmw\" (UniqueName: \"kubernetes.io/projected/130c5e5b-43ee-4329-ab13-f37b6c3981fb-kube-api-access-c5kmw\") pod \"payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl\" (UID: \"130c5e5b-43ee-4329-ab13-f37b6c3981fb\") " pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" Nov 03 20:22:30.736436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.735794 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng49b\" (UniqueName: \"kubernetes.io/projected/9e5dbdf1-f78c-4368-aabb-232578d293a7-kube-api-access-ng49b\") pod \"payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx\" (UID: \"9e5dbdf1-f78c-4368-aabb-232578d293a7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" Nov 03 20:22:30.778464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.777381 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" Nov 03 20:22:30.869699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:30.868821 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" Nov 03 20:22:31.478144 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:31.478080 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl"] Nov 03 20:22:31.570625 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:31.569672 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx"] Nov 03 20:22:32.010884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:32.010852 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" event={"ID":"9e5dbdf1-f78c-4368-aabb-232578d293a7","Type":"ContainerStarted","Data":"de36493ec97096c2941d816c9d0c5b2905711a7f834f593a9a27a7bde209baaf"} Nov 03 20:22:32.011077 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:32.010913 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" event={"ID":"9e5dbdf1-f78c-4368-aabb-232578d293a7","Type":"ContainerStarted","Data":"ceb4c2d5499a364aaf582d7f179cd4dbc5ebe9b90f71fe4f77d826443b4ba12f"} Nov 03 20:22:32.011077 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:32.010942 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" Nov 03 20:22:32.013019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:32.012987 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" event={"ID":"130c5e5b-43ee-4329-ab13-f37b6c3981fb","Type":"ContainerStarted","Data":"b3c4f49d657b41cbb403533d2a66b6d862a14b489768801f0c1b628b0fecc44f"} Nov 03 20:22:32.013165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:32.013037 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" event={"ID":"130c5e5b-43ee-4329-ab13-f37b6c3981fb","Type":"ContainerStarted","Data":"5c3bc138ea6a29d30469b7c88a1229f3e3d0cd08d16d4e27d4a12937daafce11"} Nov 03 20:22:32.013165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:32.013066 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" Nov 03 20:22:32.036798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:32.036749 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" podStartSLOduration=2.036735104 podStartE2EDuration="2.036735104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:32.03168986 +0000 UTC m=+1530.407286619" watchObservedRunningTime="2026-11-03 20:22:32.036735104 +0000 UTC m=+1530.412331850" Nov 03 20:22:32.051428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:32.051333 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" podStartSLOduration=2.051319991 podStartE2EDuration="2.051319991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:32.049371328 +0000 UTC m=+1530.424968075" watchObservedRunningTime="2026-11-03 20:22:32.051319991 +0000 UTC m=+1530.426916738" Nov 03 20:22:32.505493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:32.505461 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" Nov 03 20:22:33.515732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:33.515698 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" Nov 03 20:22:36.149195 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:22:36.149155 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:22:38.218610 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:38.218566 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4"] Nov 03 20:22:38.225265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:38.225233 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" Nov 03 20:22:38.248944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:38.235920 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-5ca474-dockercfg-p4n2l\"" Nov 03 20:22:38.248944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:38.236705 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-5ca474\"" Nov 03 20:22:38.248944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:38.237658 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4"] Nov 03 20:22:38.315411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:38.315305 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b3116a9-eaba-42f2-a977-082a8228165b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4\" (UID: \"7b3116a9-eaba-42f2-a977-082a8228165b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" Nov 03 20:22:38.316293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:38.315576 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztqmz\" (UniqueName: \"kubernetes.io/projected/7b3116a9-eaba-42f2-a977-082a8228165b-kube-api-access-ztqmz\") pod \"payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4\" (UID: \"7b3116a9-eaba-42f2-a977-082a8228165b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" Nov 03 20:22:38.441413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:38.419956 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b3116a9-eaba-42f2-a977-082a8228165b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4\" (UID: \"7b3116a9-eaba-42f2-a977-082a8228165b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" Nov 03 20:22:38.441413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:38.420248 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztqmz\" (UniqueName: \"kubernetes.io/projected/7b3116a9-eaba-42f2-a977-082a8228165b-kube-api-access-ztqmz\") pod \"payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4\" (UID: \"7b3116a9-eaba-42f2-a977-082a8228165b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" Nov 03 20:22:38.441413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:38.422316 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b3116a9-eaba-42f2-a977-082a8228165b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4\" (UID: \"7b3116a9-eaba-42f2-a977-082a8228165b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" Nov 03 20:22:38.468419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:38.465179 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztqmz\" (UniqueName: \"kubernetes.io/projected/7b3116a9-eaba-42f2-a977-082a8228165b-kube-api-access-ztqmz\") pod \"payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4\" (UID: \"7b3116a9-eaba-42f2-a977-082a8228165b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" Nov 03 20:22:38.546758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:38.546725 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" Nov 03 20:22:39.022884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:39.020686 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4"] Nov 03 20:22:39.142584 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:22:39.141468 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3116a9_eaba_42f2_a977_082a8228165b.slice/crio-188c53db6c064306333de9987b43ad6ce2c613d3432134cd2674642a3ef619a4 WatchSource:0}: Error finding container 188c53db6c064306333de9987b43ad6ce2c613d3432134cd2674642a3ef619a4: Status 404 returned error can't find the container with id 188c53db6c064306333de9987b43ad6ce2c613d3432134cd2674642a3ef619a4 Nov 03 20:22:39.304906 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:39.304866 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" event={"ID":"7b3116a9-eaba-42f2-a977-082a8228165b","Type":"ContainerStarted","Data":"188c53db6c064306333de9987b43ad6ce2c613d3432134cd2674642a3ef619a4"} Nov 03 20:22:40.316456 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:40.316417 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" event={"ID":"7b3116a9-eaba-42f2-a977-082a8228165b","Type":"ContainerStarted","Data":"88a63af6d65677da2f4fbc49d31a5461cc114e77fc55cfef874e652f59be8635"} Nov 03 20:22:40.316937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:40.316481 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" Nov 03 20:22:40.337248 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:40.337196 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" podStartSLOduration=2.337183067 podStartE2EDuration="2.337183067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:40.334166894 +0000 UTC m=+1538.709763640" watchObservedRunningTime="2026-11-03 20:22:40.337183067 +0000 UTC m=+1538.712779813" Nov 03 20:22:43.027352 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:43.027322 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" Nov 03 20:22:43.027804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:43.027410 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" Nov 03 20:22:47.940407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:47.938103 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5"] Nov 03 20:22:47.944529 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:47.944500 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:47.959362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:47.959320 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-77e83b-serving-cert\"" Nov 03 20:22:47.962615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:47.962584 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5"] Nov 03 20:22:48.059980 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.059937 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv6hx\" (UniqueName: \"kubernetes.io/projected/e8d055e5-7791-4f91-a52d-314d13289239-kube-api-access-xv6hx\") pod \"maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:48.060192 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.059996 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d055e5-7791-4f91-a52d-314d13289239-openshift-service-ca\") pod \"maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:48.060261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.060235 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-maas-api-tls\") pod \"maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:48.060317 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.060280 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-metrics-certs\") pod \"maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:48.162376 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.162337 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv6hx\" (UniqueName: \"kubernetes.io/projected/e8d055e5-7791-4f91-a52d-314d13289239-kube-api-access-xv6hx\") pod \"maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:48.162595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.162423 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d055e5-7791-4f91-a52d-314d13289239-openshift-service-ca\") pod \"maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:48.162672 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.162657 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-maas-api-tls\") pod \"maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:48.162718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.162703 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-metrics-certs\") pod \"maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:48.163892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.163846 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d055e5-7791-4f91-a52d-314d13289239-openshift-service-ca\") pod \"maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:48.166657 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.166634 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-metrics-certs\") pod \"maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:48.168151 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.168126 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-maas-api-tls\") pod \"maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:48.176297 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.176267 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv6hx\" (UniqueName: \"kubernetes.io/projected/e8d055e5-7791-4f91-a52d-314d13289239-kube-api-access-xv6hx\") pod \"maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:48.277103 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.277005 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:48.767412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.766018 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5"] Nov 03 20:22:48.775411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:48.771104 4457 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:22:49.509740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:49.509657 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" event={"ID":"e8d055e5-7791-4f91-a52d-314d13289239","Type":"ContainerStarted","Data":"88883fc000b94f46e32228a4017f26c61fa9138cae7fcda5b4b5f88ec8593a6f"} Nov 03 20:22:49.510207 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:49.510188 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" event={"ID":"e8d055e5-7791-4f91-a52d-314d13289239","Type":"ContainerStarted","Data":"726b859e045dfddf16b6c0a6be918166b3c0ef6e3b71628577ec8ea9f658fbf9"} Nov 03 20:22:49.510329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:49.510319 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:49.532166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:49.532100 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" podStartSLOduration=2.110750805 podStartE2EDuration="2.532082603s" totalImagesPullingTime="421.331798ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:49.529185454 +0000 UTC m=+1547.904782202" watchObservedRunningTime="2026-11-03 20:22:49.532082603 +0000 UTC m=+1547.907679352" Nov 03 20:22:51.237771 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:51.237716 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb"] Nov 03 20:22:51.248243 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:51.248197 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d5927-hwwtb"] Nov 03 20:22:51.329696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:51.329665 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" Nov 03 20:22:51.760242 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:51.760208 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8037473-08c1-4d3c-a9d6-8d61810da17f" path="/var/lib/kubelet/pods/c8037473-08c1-4d3c-a9d6-8d61810da17f/volumes" Nov 03 20:22:52.158285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.158249 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq"] Nov 03 20:22:52.165318 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.165295 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" Nov 03 20:22:52.174705 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.174676 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:22:52.174861 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.174725 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:22:52.176114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.176089 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq"] Nov 03 20:22:52.292117 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.292082 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq\" (UID: \"eeda8067-4cfe-482f-902e-c1c62dc46040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" Nov 03 20:22:52.292573 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.292309 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq\" (UID: \"eeda8067-4cfe-482f-902e-c1c62dc46040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" Nov 03 20:22:52.292573 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.292418 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkpvz\" (UniqueName: \"kubernetes.io/projected/eeda8067-4cfe-482f-902e-c1c62dc46040-kube-api-access-qkpvz\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq\" (UID: \"eeda8067-4cfe-482f-902e-c1c62dc46040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" Nov 03 20:22:52.394142 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.394105 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq\" (UID: \"eeda8067-4cfe-482f-902e-c1c62dc46040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" Nov 03 20:22:52.394377 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.394352 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq\" (UID: \"eeda8067-4cfe-482f-902e-c1c62dc46040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" Nov 03 20:22:52.394543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.394526 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkpvz\" (UniqueName: \"kubernetes.io/projected/eeda8067-4cfe-482f-902e-c1c62dc46040-kube-api-access-qkpvz\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq\" (UID: \"eeda8067-4cfe-482f-902e-c1c62dc46040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" Nov 03 20:22:52.395077 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.395051 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq\" (UID: \"eeda8067-4cfe-482f-902e-c1c62dc46040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" Nov 03 20:22:52.396861 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.396830 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq\" (UID: \"eeda8067-4cfe-482f-902e-c1c62dc46040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" Nov 03 20:22:52.407348 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.407321 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkpvz\" (UniqueName: \"kubernetes.io/projected/eeda8067-4cfe-482f-902e-c1c62dc46040-kube-api-access-qkpvz\") pod \"maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq\" (UID: \"eeda8067-4cfe-482f-902e-c1c62dc46040\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" Nov 03 20:22:52.488566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.488480 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" Nov 03 20:22:52.810464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:52.809444 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq"] Nov 03 20:22:53.571091 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:53.571048 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" event={"ID":"eeda8067-4cfe-482f-902e-c1c62dc46040","Type":"ContainerStarted","Data":"ae44e10270bce3c2c7a3fbe2690b29b4e8ab2b9d32d6c44c40708c8b87ea96eb"} Nov 03 20:22:53.571527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:53.571117 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" event={"ID":"eeda8067-4cfe-482f-902e-c1c62dc46040","Type":"ContainerStarted","Data":"990064cb4745e06415084988ed10f19c4f2ac88a849fb9f308bf82cb983465e2"} Nov 03 20:22:53.599589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:53.599532 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" podStartSLOduration=1.599519125 podStartE2EDuration="1.599519125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:22:53.59502938 +0000 UTC m=+1551.970626122" watchObservedRunningTime="2026-11-03 20:22:53.599519125 +0000 UTC m=+1551.975115873" Nov 03 20:22:54.583104 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:54.583065 4457 generic.go:356] "Generic (PLEG): container finished" podID="eeda8067-4cfe-482f-902e-c1c62dc46040" containerID="ae44e10270bce3c2c7a3fbe2690b29b4e8ab2b9d32d6c44c40708c8b87ea96eb" exitCode=0 Nov 03 20:22:54.583665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:54.583153 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" event={"ID":"eeda8067-4cfe-482f-902e-c1c62dc46040","Type":"ContainerDied","Data":"ae44e10270bce3c2c7a3fbe2690b29b4e8ab2b9d32d6c44c40708c8b87ea96eb"} Nov 03 20:22:55.839153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:55.839127 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" Nov 03 20:22:55.974910 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:55.974805 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-xks-ca\") pod \"eeda8067-4cfe-482f-902e-c1c62dc46040\" (UID: \"eeda8067-4cfe-482f-902e-c1c62dc46040\") " Nov 03 20:22:55.975117 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:55.975096 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeda8067-4cfe-482f-902e-c1c62dc46040-kube-api-access-qkpvz\" (UniqueName: \"kubernetes.io/projected/eeda8067-4cfe-482f-902e-c1c62dc46040-kube-api-access-qkpvz\") pod \"eeda8067-4cfe-482f-902e-c1c62dc46040\" (UID: \"eeda8067-4cfe-482f-902e-c1c62dc46040\") " Nov 03 20:22:55.975186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:55.975138 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-openshift-ca\") pod \"eeda8067-4cfe-482f-902e-c1c62dc46040\" (UID: \"eeda8067-4cfe-482f-902e-c1c62dc46040\") " Nov 03 20:22:55.975586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:55.975556 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-openshift-ca" pod "eeda8067-4cfe-482f-902e-c1c62dc46040" (UID: "eeda8067-4cfe-482f-902e-c1c62dc46040"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:22:55.975943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:55.975924 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:55.976939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:55.976911 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-xks-ca" pod "eeda8067-4cfe-482f-902e-c1c62dc46040" (UID: "eeda8067-4cfe-482f-902e-c1c62dc46040"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:22:55.977424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:55.977372 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeda8067-4cfe-482f-902e-c1c62dc46040-kube-api-access-qkpvz" pod "eeda8067-4cfe-482f-902e-c1c62dc46040" (UID: "eeda8067-4cfe-482f-902e-c1c62dc46040"). InnerVolumeSpecName "kube-api-access-qkpvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:22:56.076975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:56.076947 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeda8067-4cfe-482f-902e-c1c62dc46040-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:56.076975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:56.076975 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkpvz\" (UniqueName: \"kubernetes.io/projected/eeda8067-4cfe-482f-902e-c1c62dc46040-kube-api-access-qkpvz\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:22:56.542245 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:56.542215 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:22:56.609367 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:56.609324 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" event={"ID":"eeda8067-4cfe-482f-902e-c1c62dc46040","Type":"ContainerDied","Data":"990064cb4745e06415084988ed10f19c4f2ac88a849fb9f308bf82cb983465e2"} Nov 03 20:22:56.609590 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:56.609441 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="990064cb4745e06415084988ed10f19c4f2ac88a849fb9f308bf82cb983465e2" Nov 03 20:22:56.609590 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:56.609521 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq" Nov 03 20:22:59.486317 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.486274 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f"] Nov 03 20:22:59.487511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.487487 4457 cpu_manager.go:414] "containerMap: removing container" podUID="eeda8067-4cfe-482f-902e-c1c62dc46040" containerName="revoke-keys" Nov 03 20:22:59.487511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.487510 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeda8067-4cfe-482f-902e-c1c62dc46040" containerName="revoke-keys" Nov 03 20:22:59.487770 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.487757 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeda8067-4cfe-482f-902e-c1c62dc46040" containerName="revoke-keys" Nov 03 20:22:59.492714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.492691 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" Nov 03 20:22:59.500261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.500237 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f"] Nov 03 20:22:59.565115 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.565078 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:22:59.565284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.565186 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:22:59.600238 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.600198 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f\" (UID: \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" Nov 03 20:22:59.600461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.600441 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5zp7\" (UniqueName: \"kubernetes.io/projected/1b0f7519-da9f-4e59-a8ae-64d16862eef2-kube-api-access-x5zp7\") pod \"maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f\" (UID: \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" Nov 03 20:22:59.600528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.600519 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f\" (UID: \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" Nov 03 20:22:59.702426 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.702374 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f\" (UID: \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" Nov 03 20:22:59.702606 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.702565 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5zp7\" (UniqueName: \"kubernetes.io/projected/1b0f7519-da9f-4e59-a8ae-64d16862eef2-kube-api-access-x5zp7\") pod \"maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f\" (UID: \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" Nov 03 20:22:59.702675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.702635 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f\" (UID: \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" Nov 03 20:22:59.703348 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.703321 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f\" (UID: \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" Nov 03 20:22:59.704955 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.704932 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f\" (UID: \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" Nov 03 20:22:59.714890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.714862 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5zp7\" (UniqueName: \"kubernetes.io/projected/1b0f7519-da9f-4e59-a8ae-64d16862eef2-kube-api-access-x5zp7\") pod \"maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f\" (UID: \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" Nov 03 20:22:59.824411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:22:59.821756 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" Nov 03 20:23:00.309272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:00.309230 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f"] Nov 03 20:23:00.646362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:00.646271 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" event={"ID":"1b0f7519-da9f-4e59-a8ae-64d16862eef2","Type":"ContainerStarted","Data":"147de55b30cd5b3d30726a5fba9e1334432394b2b69cff1944f42268077e78f3"} Nov 03 20:23:01.657584 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:01.657549 4457 generic.go:356] "Generic (PLEG): container finished" podID="1b0f7519-da9f-4e59-a8ae-64d16862eef2" containerID="8397124c8293e5020d581883c0b390180d79f2393be929d28f2deb06e331f4e9" exitCode=0 Nov 03 20:23:01.658001 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:01.657640 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" event={"ID":"1b0f7519-da9f-4e59-a8ae-64d16862eef2","Type":"ContainerDied","Data":"8397124c8293e5020d581883c0b390180d79f2393be929d28f2deb06e331f4e9"} Nov 03 20:23:02.308818 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:02.308785 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77"] Nov 03 20:23:02.321436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:02.321369 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4"] Nov 03 20:23:03.326085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:03.326043 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" Nov 03 20:23:03.422595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:03.422564 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b0f7519-da9f-4e59-a8ae-64d16862eef2-kube-api-access-x5zp7\" (UniqueName: \"kubernetes.io/projected/1b0f7519-da9f-4e59-a8ae-64d16862eef2-kube-api-access-x5zp7\") pod \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\" (UID: \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\") " Nov 03 20:23:03.422794 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:03.422623 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-openshift-ca\") pod \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\" (UID: \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\") " Nov 03 20:23:03.422794 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:03.422656 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-xks-ca\") pod \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\" (UID: \"1b0f7519-da9f-4e59-a8ae-64d16862eef2\") " Nov 03 20:23:03.423080 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:03.423053 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-openshift-ca" pod "1b0f7519-da9f-4e59-a8ae-64d16862eef2" (UID: "1b0f7519-da9f-4e59-a8ae-64d16862eef2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:03.423548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:03.423524 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:03.424791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:03.424762 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-xks-ca" pod "1b0f7519-da9f-4e59-a8ae-64d16862eef2" (UID: "1b0f7519-da9f-4e59-a8ae-64d16862eef2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:23:03.424791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:03.424766 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0f7519-da9f-4e59-a8ae-64d16862eef2-kube-api-access-x5zp7" pod "1b0f7519-da9f-4e59-a8ae-64d16862eef2" (UID: "1b0f7519-da9f-4e59-a8ae-64d16862eef2"). InnerVolumeSpecName "kube-api-access-x5zp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:03.525371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:03.525337 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5zp7\" (UniqueName: \"kubernetes.io/projected/1b0f7519-da9f-4e59-a8ae-64d16862eef2-kube-api-access-x5zp7\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:03.525371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:03.525370 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b0f7519-da9f-4e59-a8ae-64d16862eef2-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:03.707114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:03.707029 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" Nov 03 20:23:03.707114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:03.707060 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f" event={"ID":"1b0f7519-da9f-4e59-a8ae-64d16862eef2","Type":"ContainerDied","Data":"147de55b30cd5b3d30726a5fba9e1334432394b2b69cff1944f42268077e78f3"} Nov 03 20:23:03.707114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:03.707105 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="147de55b30cd5b3d30726a5fba9e1334432394b2b69cff1944f42268077e78f3" Nov 03 20:23:06.147259 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:23:06.147225 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:23:06.413848 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:06.413759 4457 scope.go:122] "RemoveContainer" containerID="0efbd5099d4517aa88482212252a70178fe46c5312f25c4fcfd7403d59162fc6" Nov 03 20:23:06.425923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:06.425903 4457 scope.go:122] "RemoveContainer" containerID="ce1d8b7d61bf2a99c1db87f1402ae09b18417fecc172d7aaed043f9e51aee184" Nov 03 20:23:06.442362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:06.442335 4457 scope.go:122] "RemoveContainer" containerID="46dce2e31628953bfe1a9bb7d330a5b0ec2e177aafb8c135a4da4d0fc39f1e72" Nov 03 20:23:07.309462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.309423 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" podUID="4c9fe6d7-437e-4ef6-97dd-22e7138b8690" containerName="payload-processing" containerID="cri-o://ab3948abd4ace9e506bc5ba0c4e3ee9e1d52080449ba7530e6cc85bbfe69d11b" gracePeriod=25 Nov 03 20:23:07.322555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.322514 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" podUID="52bb2310-0cce-4dc0-97ac-484f919f099c" containerName="payload-pre-processing" containerID="cri-o://b4ec95391cff73ab8485d7a5a653b5dce82058d25ed12524321d8a6ddd2055a7" gracePeriod=25 Nov 03 20:23:07.657205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.657172 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" Nov 03 20:23:07.669604 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.669568 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" Nov 03 20:23:07.732716 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.732672 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-plugins-config-volume\") pod \"4c9fe6d7-437e-4ef6-97dd-22e7138b8690\" (UID: \"4c9fe6d7-437e-4ef6-97dd-22e7138b8690\") " Nov 03 20:23:07.732933 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.732828 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-kube-api-access-m5qxh\" (UniqueName: \"kubernetes.io/projected/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-kube-api-access-m5qxh\") pod \"4c9fe6d7-437e-4ef6-97dd-22e7138b8690\" (UID: \"4c9fe6d7-437e-4ef6-97dd-22e7138b8690\") " Nov 03 20:23:07.732933 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.732908 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52bb2310-0cce-4dc0-97ac-484f919f099c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bb2310-0cce-4dc0-97ac-484f919f099c-plugins-config-volume\") pod \"52bb2310-0cce-4dc0-97ac-484f919f099c\" (UID: \"52bb2310-0cce-4dc0-97ac-484f919f099c\") " Nov 03 20:23:07.733130 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.733109 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52bb2310-0cce-4dc0-97ac-484f919f099c-kube-api-access-nx2rh\" (UniqueName: \"kubernetes.io/projected/52bb2310-0cce-4dc0-97ac-484f919f099c-kube-api-access-nx2rh\") pod \"52bb2310-0cce-4dc0-97ac-484f919f099c\" (UID: \"52bb2310-0cce-4dc0-97ac-484f919f099c\") " Nov 03 20:23:07.743634 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.743082 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52bb2310-0cce-4dc0-97ac-484f919f099c-plugins-config-volume" pod "52bb2310-0cce-4dc0-97ac-484f919f099c" (UID: "52bb2310-0cce-4dc0-97ac-484f919f099c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:07.744521 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.744492 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-plugins-config-volume" pod "4c9fe6d7-437e-4ef6-97dd-22e7138b8690" (UID: "4c9fe6d7-437e-4ef6-97dd-22e7138b8690"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:07.791649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.790903 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-kube-api-access-m5qxh" pod "4c9fe6d7-437e-4ef6-97dd-22e7138b8690" (UID: "4c9fe6d7-437e-4ef6-97dd-22e7138b8690"). InnerVolumeSpecName "kube-api-access-m5qxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:07.792802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.792630 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bb2310-0cce-4dc0-97ac-484f919f099c-kube-api-access-nx2rh" pod "52bb2310-0cce-4dc0-97ac-484f919f099c" (UID: "52bb2310-0cce-4dc0-97ac-484f919f099c"). InnerVolumeSpecName "kube-api-access-nx2rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:07.828735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.828677 4457 generic.go:356] "Generic (PLEG): container finished" podID="4c9fe6d7-437e-4ef6-97dd-22e7138b8690" containerID="ab3948abd4ace9e506bc5ba0c4e3ee9e1d52080449ba7530e6cc85bbfe69d11b" exitCode=0 Nov 03 20:23:07.828887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.828735 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" Nov 03 20:23:07.828887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.828749 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" event={"ID":"4c9fe6d7-437e-4ef6-97dd-22e7138b8690","Type":"ContainerDied","Data":"ab3948abd4ace9e506bc5ba0c4e3ee9e1d52080449ba7530e6cc85bbfe69d11b"} Nov 03 20:23:07.828887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.828805 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77" event={"ID":"4c9fe6d7-437e-4ef6-97dd-22e7138b8690","Type":"ContainerDied","Data":"ceb7eac8f5168503858cdf7120feed60c015b049d89f72c5f65d988acccbddd1"} Nov 03 20:23:07.828887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.828829 4457 scope.go:122] "RemoveContainer" containerID="ab3948abd4ace9e506bc5ba0c4e3ee9e1d52080449ba7530e6cc85bbfe69d11b" Nov 03 20:23:07.831633 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.831609 4457 generic.go:356] "Generic (PLEG): container finished" podID="52bb2310-0cce-4dc0-97ac-484f919f099c" containerID="b4ec95391cff73ab8485d7a5a653b5dce82058d25ed12524321d8a6ddd2055a7" exitCode=0 Nov 03 20:23:07.831780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.831677 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" event={"ID":"52bb2310-0cce-4dc0-97ac-484f919f099c","Type":"ContainerDied","Data":"b4ec95391cff73ab8485d7a5a653b5dce82058d25ed12524321d8a6ddd2055a7"} Nov 03 20:23:07.831780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.831717 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" event={"ID":"52bb2310-0cce-4dc0-97ac-484f919f099c","Type":"ContainerDied","Data":"ed39122fd0c82b71405e8c5b541bed85c4b9729de9046ac7deece986af167f9b"} Nov 03 20:23:07.832013 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.831999 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4" Nov 03 20:23:07.836117 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.836093 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx2rh\" (UniqueName: \"kubernetes.io/projected/52bb2310-0cce-4dc0-97ac-484f919f099c-kube-api-access-nx2rh\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:07.836211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.836122 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:07.836211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.836138 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qxh\" (UniqueName: \"kubernetes.io/projected/4c9fe6d7-437e-4ef6-97dd-22e7138b8690-kube-api-access-m5qxh\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:07.836211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.836153 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52bb2310-0cce-4dc0-97ac-484f919f099c-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:07.851253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.851226 4457 scope.go:122] "RemoveContainer" containerID="ab3948abd4ace9e506bc5ba0c4e3ee9e1d52080449ba7530e6cc85bbfe69d11b" Nov 03 20:23:07.852718 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:23:07.852685 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3948abd4ace9e506bc5ba0c4e3ee9e1d52080449ba7530e6cc85bbfe69d11b\": container with ID starting with ab3948abd4ace9e506bc5ba0c4e3ee9e1d52080449ba7530e6cc85bbfe69d11b not found: ID does not exist" containerID="ab3948abd4ace9e506bc5ba0c4e3ee9e1d52080449ba7530e6cc85bbfe69d11b" Nov 03 20:23:07.852823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.852732 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3948abd4ace9e506bc5ba0c4e3ee9e1d52080449ba7530e6cc85bbfe69d11b"} err="failed to get container status \"ab3948abd4ace9e506bc5ba0c4e3ee9e1d52080449ba7530e6cc85bbfe69d11b\": rpc error: code = NotFound desc = could not find container \"ab3948abd4ace9e506bc5ba0c4e3ee9e1d52080449ba7530e6cc85bbfe69d11b\": container with ID starting with ab3948abd4ace9e506bc5ba0c4e3ee9e1d52080449ba7530e6cc85bbfe69d11b not found: ID does not exist" Nov 03 20:23:07.852823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.852759 4457 scope.go:122] "RemoveContainer" containerID="b4ec95391cff73ab8485d7a5a653b5dce82058d25ed12524321d8a6ddd2055a7" Nov 03 20:23:07.868743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.868723 4457 scope.go:122] "RemoveContainer" containerID="b4ec95391cff73ab8485d7a5a653b5dce82058d25ed12524321d8a6ddd2055a7" Nov 03 20:23:07.869144 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:23:07.869117 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4ec95391cff73ab8485d7a5a653b5dce82058d25ed12524321d8a6ddd2055a7\": container with ID starting with b4ec95391cff73ab8485d7a5a653b5dce82058d25ed12524321d8a6ddd2055a7 not found: ID does not exist" containerID="b4ec95391cff73ab8485d7a5a653b5dce82058d25ed12524321d8a6ddd2055a7" Nov 03 20:23:07.869251 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.869153 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4ec95391cff73ab8485d7a5a653b5dce82058d25ed12524321d8a6ddd2055a7"} err="failed to get container status \"b4ec95391cff73ab8485d7a5a653b5dce82058d25ed12524321d8a6ddd2055a7\": rpc error: code = NotFound desc = could not find container \"b4ec95391cff73ab8485d7a5a653b5dce82058d25ed12524321d8a6ddd2055a7\": container with ID starting with b4ec95391cff73ab8485d7a5a653b5dce82058d25ed12524321d8a6ddd2055a7 not found: ID does not exist" Nov 03 20:23:07.880259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.879878 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77"] Nov 03 20:23:07.888905 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.888878 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-15e02e6b-65c4d8d68d-44r77"] Nov 03 20:23:07.896963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.896942 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4"] Nov 03 20:23:07.901893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:07.901874 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-15e02e6b-7647759-j7tb4"] Nov 03 20:23:09.763308 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:09.763271 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c9fe6d7-437e-4ef6-97dd-22e7138b8690" path="/var/lib/kubelet/pods/4c9fe6d7-437e-4ef6-97dd-22e7138b8690/volumes" Nov 03 20:23:09.764059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:09.764024 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52bb2310-0cce-4dc0-97ac-484f919f099c" path="/var/lib/kubelet/pods/52bb2310-0cce-4dc0-97ac-484f919f099c/volumes" Nov 03 20:23:12.431072 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:12.427684 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq"] Nov 03 20:23:12.433917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:12.433878 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-15e02e6b-rbpfq"] Nov 03 20:23:13.792727 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:13.792687 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeda8067-4cfe-482f-902e-c1c62dc46040" path="/var/lib/kubelet/pods/eeda8067-4cfe-482f-902e-c1c62dc46040/volumes" Nov 03 20:23:17.616222 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:17.616027 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg"] Nov 03 20:23:17.616860 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:17.616824 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" podUID="5d8658c4-9961-4acd-8180-20128324c476" containerName="istio-proxy" containerID="cri-o://dacb3def3c367b61616568cc0dd63b85354b96c33d7249e3414d1cdd8f9ed201" gracePeriod=30 Nov 03 20:23:22.932152 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:22.932123 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:23:23.006993 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.006917 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-certs\") pod \"5d8658c4-9961-4acd-8180-20128324c476\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " Nov 03 20:23:23.007191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.006999 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d8658c4-9961-4acd-8180-20128324c476-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8658c4-9961-4acd-8180-20128324c476-istiod-ca-cert\") pod \"5d8658c4-9961-4acd-8180-20128324c476\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " Nov 03 20:23:23.007191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.007111 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d8658c4-9961-4acd-8180-20128324c476-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8658c4-9961-4acd-8180-20128324c476-istio-podinfo\") pod \"5d8658c4-9961-4acd-8180-20128324c476\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " Nov 03 20:23:23.007191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.007182 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-istio-token\" (UniqueName: \"kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-istio-token\") pod \"5d8658c4-9961-4acd-8180-20128324c476\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " Nov 03 20:23:23.007442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.007235 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-data\") pod \"5d8658c4-9961-4acd-8180-20128324c476\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " Nov 03 20:23:23.007442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.007226 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-certs" pod "5d8658c4-9961-4acd-8180-20128324c476" (UID: "5d8658c4-9961-4acd-8180-20128324c476"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:23.007442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.007296 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-envoy\") pod \"5d8658c4-9961-4acd-8180-20128324c476\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " Nov 03 20:23:23.007442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.007336 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-socket\") pod \"5d8658c4-9961-4acd-8180-20128324c476\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " Nov 03 20:23:23.007442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.007407 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8658c4-9961-4acd-8180-20128324c476-istiod-ca-cert" pod "5d8658c4-9961-4acd-8180-20128324c476" (UID: "5d8658c4-9961-4acd-8180-20128324c476"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:23:23.007736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.007488 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-kube-api-access-rdkb4\" (UniqueName: \"kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-kube-api-access-rdkb4\") pod \"5d8658c4-9961-4acd-8180-20128324c476\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " Nov 03 20:23:23.007736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.007546 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-credential-socket\") pod \"5d8658c4-9961-4acd-8180-20128324c476\" (UID: \"5d8658c4-9961-4acd-8180-20128324c476\") " Nov 03 20:23:23.007736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.007665 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-data" pod "5d8658c4-9961-4acd-8180-20128324c476" (UID: "5d8658c4-9961-4acd-8180-20128324c476"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:23.008154 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.008131 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-socket" pod "5d8658c4-9961-4acd-8180-20128324c476" (UID: "5d8658c4-9961-4acd-8180-20128324c476"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:23.008531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.008507 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:23.008630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.008557 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d8658c4-9961-4acd-8180-20128324c476-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:23.008630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.008574 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:23.009003 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.008959 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-credential-socket" pod "5d8658c4-9961-4acd-8180-20128324c476" (UID: "5d8658c4-9961-4acd-8180-20128324c476"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:23.009930 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.009905 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d8658c4-9961-4acd-8180-20128324c476-istio-podinfo" pod "5d8658c4-9961-4acd-8180-20128324c476" (UID: "5d8658c4-9961-4acd-8180-20128324c476"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:23:23.010078 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.010059 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-istio-token" pod "5d8658c4-9961-4acd-8180-20128324c476" (UID: "5d8658c4-9961-4acd-8180-20128324c476"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:23.010179 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.010163 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-envoy" pod "5d8658c4-9961-4acd-8180-20128324c476" (UID: "5d8658c4-9961-4acd-8180-20128324c476"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:23:23.011486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.011464 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-kube-api-access-rdkb4" pod "5d8658c4-9961-4acd-8180-20128324c476" (UID: "5d8658c4-9961-4acd-8180-20128324c476"). InnerVolumeSpecName "kube-api-access-rdkb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:23:23.110572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.110537 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:23.110572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.110564 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d8658c4-9961-4acd-8180-20128324c476-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:23.110572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.110574 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:23.110812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.110588 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:23.110812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.110598 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d8658c4-9961-4acd-8180-20128324c476-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:23.110812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.110607 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdkb4\" (UniqueName: \"kubernetes.io/projected/5d8658c4-9961-4acd-8180-20128324c476-kube-api-access-rdkb4\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:23:23.453492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.453453 4457 generic.go:356] "Generic (PLEG): container finished" podID="5d8658c4-9961-4acd-8180-20128324c476" containerID="dacb3def3c367b61616568cc0dd63b85354b96c33d7249e3414d1cdd8f9ed201" exitCode=0 Nov 03 20:23:23.453712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.453509 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" Nov 03 20:23:23.453712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.453534 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" event={"ID":"5d8658c4-9961-4acd-8180-20128324c476","Type":"ContainerDied","Data":"dacb3def3c367b61616568cc0dd63b85354b96c33d7249e3414d1cdd8f9ed201"} Nov 03 20:23:23.453712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.453594 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg" event={"ID":"5d8658c4-9961-4acd-8180-20128324c476","Type":"ContainerDied","Data":"e08ef4544bb40aa4c8cb63efc811457bfae1ad6022bfddc206137aa115898aa1"} Nov 03 20:23:23.453712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.453618 4457 scope.go:122] "RemoveContainer" containerID="dacb3def3c367b61616568cc0dd63b85354b96c33d7249e3414d1cdd8f9ed201" Nov 03 20:23:23.466902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.466879 4457 scope.go:122] "RemoveContainer" containerID="dacb3def3c367b61616568cc0dd63b85354b96c33d7249e3414d1cdd8f9ed201" Nov 03 20:23:23.467232 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:23:23.467182 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacb3def3c367b61616568cc0dd63b85354b96c33d7249e3414d1cdd8f9ed201\": container with ID starting with dacb3def3c367b61616568cc0dd63b85354b96c33d7249e3414d1cdd8f9ed201 not found: ID does not exist" containerID="dacb3def3c367b61616568cc0dd63b85354b96c33d7249e3414d1cdd8f9ed201" Nov 03 20:23:23.467340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.467228 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacb3def3c367b61616568cc0dd63b85354b96c33d7249e3414d1cdd8f9ed201"} err="failed to get container status \"dacb3def3c367b61616568cc0dd63b85354b96c33d7249e3414d1cdd8f9ed201\": rpc error: code = NotFound desc = could not find container \"dacb3def3c367b61616568cc0dd63b85354b96c33d7249e3414d1cdd8f9ed201\": container with ID starting with dacb3def3c367b61616568cc0dd63b85354b96c33d7249e3414d1cdd8f9ed201 not found: ID does not exist" Nov 03 20:23:23.483824 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.483796 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg"] Nov 03 20:23:23.490718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.490694 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-rwcpg"] Nov 03 20:23:23.763627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:23.763546 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d8658c4-9961-4acd-8180-20128324c476" path="/var/lib/kubelet/pods/5d8658c4-9961-4acd-8180-20128324c476/volumes" Nov 03 20:23:24.238926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.238892 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j"] Nov 03 20:23:24.239715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.239698 4457 cpu_manager.go:414] "containerMap: removing container" podUID="4c9fe6d7-437e-4ef6-97dd-22e7138b8690" containerName="payload-processing" Nov 03 20:23:24.239715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.239715 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c9fe6d7-437e-4ef6-97dd-22e7138b8690" containerName="payload-processing" Nov 03 20:23:24.239868 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.239730 4457 cpu_manager.go:414] "containerMap: removing container" podUID="5d8658c4-9961-4acd-8180-20128324c476" containerName="istio-proxy" Nov 03 20:23:24.239868 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.239735 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d8658c4-9961-4acd-8180-20128324c476" containerName="istio-proxy" Nov 03 20:23:24.239868 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.239747 4457 cpu_manager.go:414] "containerMap: removing container" podUID="1b0f7519-da9f-4e59-a8ae-64d16862eef2" containerName="revoke-keys" Nov 03 20:23:24.239868 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.239752 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b0f7519-da9f-4e59-a8ae-64d16862eef2" containerName="revoke-keys" Nov 03 20:23:24.239868 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.239771 4457 cpu_manager.go:414] "containerMap: removing container" podUID="52bb2310-0cce-4dc0-97ac-484f919f099c" containerName="payload-pre-processing" Nov 03 20:23:24.239868 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.239777 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52bb2310-0cce-4dc0-97ac-484f919f099c" containerName="payload-pre-processing" Nov 03 20:23:24.240170 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.239876 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="52bb2310-0cce-4dc0-97ac-484f919f099c" containerName="payload-pre-processing" Nov 03 20:23:24.240170 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.239893 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c9fe6d7-437e-4ef6-97dd-22e7138b8690" containerName="payload-processing" Nov 03 20:23:24.240170 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.239901 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d8658c4-9961-4acd-8180-20128324c476" containerName="istio-proxy" Nov 03 20:23:24.240170 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.239907 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b0f7519-da9f-4e59-a8ae-64d16862eef2" containerName="revoke-keys" Nov 03 20:23:24.243747 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.243728 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" Nov 03 20:23:24.254723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.254696 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-34e844\"" Nov 03 20:23:24.262477 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.262448 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j"] Nov 03 20:23:24.316265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.316226 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv"] Nov 03 20:23:24.321243 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.321219 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" Nov 03 20:23:24.334129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.334107 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-34e844-dockercfg-njpp7\"" Nov 03 20:23:24.339612 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.339585 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv"] Nov 03 20:23:24.340217 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.340188 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899751b-58b7-428e-a0b1-464a005dfabe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j\" (UID: \"9899751b-58b7-428e-a0b1-464a005dfabe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" Nov 03 20:23:24.340470 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.340441 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqhb9\" (UniqueName: \"kubernetes.io/projected/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-kube-api-access-qqhb9\") pod \"payload-processing-e2e-inf-b-34e844-599b659bb-b52jv\" (UID: \"2243a6fb-07fb-4f03-ba5f-76d5909e7a44\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" Nov 03 20:23:24.340637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.340619 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sm4b\" (UniqueName: \"kubernetes.io/projected/9899751b-58b7-428e-a0b1-464a005dfabe-kube-api-access-7sm4b\") pod \"payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j\" (UID: \"9899751b-58b7-428e-a0b1-464a005dfabe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" Nov 03 20:23:24.340778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.340762 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-34e844-599b659bb-b52jv\" (UID: \"2243a6fb-07fb-4f03-ba5f-76d5909e7a44\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" Nov 03 20:23:24.488412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.451438 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899751b-58b7-428e-a0b1-464a005dfabe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j\" (UID: \"9899751b-58b7-428e-a0b1-464a005dfabe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" Nov 03 20:23:24.488412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.451597 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqhb9\" (UniqueName: \"kubernetes.io/projected/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-kube-api-access-qqhb9\") pod \"payload-processing-e2e-inf-b-34e844-599b659bb-b52jv\" (UID: \"2243a6fb-07fb-4f03-ba5f-76d5909e7a44\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" Nov 03 20:23:24.488412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.451812 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sm4b\" (UniqueName: \"kubernetes.io/projected/9899751b-58b7-428e-a0b1-464a005dfabe-kube-api-access-7sm4b\") pod \"payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j\" (UID: \"9899751b-58b7-428e-a0b1-464a005dfabe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" Nov 03 20:23:24.488412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.471231 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-34e844-599b659bb-b52jv\" (UID: \"2243a6fb-07fb-4f03-ba5f-76d5909e7a44\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" Nov 03 20:23:24.488412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.472317 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-34e844-599b659bb-b52jv\" (UID: \"2243a6fb-07fb-4f03-ba5f-76d5909e7a44\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" Nov 03 20:23:24.488412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.474411 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899751b-58b7-428e-a0b1-464a005dfabe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j\" (UID: \"9899751b-58b7-428e-a0b1-464a005dfabe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" Nov 03 20:23:24.537810 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.522048 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sm4b\" (UniqueName: \"kubernetes.io/projected/9899751b-58b7-428e-a0b1-464a005dfabe-kube-api-access-7sm4b\") pod \"payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j\" (UID: \"9899751b-58b7-428e-a0b1-464a005dfabe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" Nov 03 20:23:24.591676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.591642 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqhb9\" (UniqueName: \"kubernetes.io/projected/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-kube-api-access-qqhb9\") pod \"payload-processing-e2e-inf-b-34e844-599b659bb-b52jv\" (UID: \"2243a6fb-07fb-4f03-ba5f-76d5909e7a44\") " pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" Nov 03 20:23:24.593559 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.593526 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" Nov 03 20:23:24.643409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:24.641949 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" Nov 03 20:23:25.010614 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:23:25.010586 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2243a6fb_07fb_4f03_ba5f_76d5909e7a44.slice/crio-c1febcefa3dca3d77c86dbc976d3859f2fca045df553517650950da6d46c0513 WatchSource:0}: Error finding container c1febcefa3dca3d77c86dbc976d3859f2fca045df553517650950da6d46c0513: Status 404 returned error can't find the container with id c1febcefa3dca3d77c86dbc976d3859f2fca045df553517650950da6d46c0513 Nov 03 20:23:25.011659 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:25.011632 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv"] Nov 03 20:23:25.149526 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:23:25.149496 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9899751b_58b7_428e_a0b1_464a005dfabe.slice/crio-4f7e8a307327f8aec6f8c87492386c9390adfd42ce75ae608cc79a8da9290d9d WatchSource:0}: Error finding container 4f7e8a307327f8aec6f8c87492386c9390adfd42ce75ae608cc79a8da9290d9d: Status 404 returned error can't find the container with id 4f7e8a307327f8aec6f8c87492386c9390adfd42ce75ae608cc79a8da9290d9d Nov 03 20:23:25.151516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:25.151491 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j"] Nov 03 20:23:25.510251 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:25.510140 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" event={"ID":"9899751b-58b7-428e-a0b1-464a005dfabe","Type":"ContainerStarted","Data":"863ec92fefd1eb2585e90128af0770cfe26667573146e522acb5356a215cdf7d"} Nov 03 20:23:25.510251 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:25.510220 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" Nov 03 20:23:25.510251 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:25.510246 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" event={"ID":"9899751b-58b7-428e-a0b1-464a005dfabe","Type":"ContainerStarted","Data":"4f7e8a307327f8aec6f8c87492386c9390adfd42ce75ae608cc79a8da9290d9d"} Nov 03 20:23:25.512060 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:25.512032 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" event={"ID":"2243a6fb-07fb-4f03-ba5f-76d5909e7a44","Type":"ContainerStarted","Data":"54e904e1ab51e4388649c5f4e37c16d47dcaaba31be19c4eb81d66e09d874b5b"} Nov 03 20:23:25.512192 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:25.512089 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" Nov 03 20:23:25.512192 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:25.512114 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" event={"ID":"2243a6fb-07fb-4f03-ba5f-76d5909e7a44","Type":"ContainerStarted","Data":"c1febcefa3dca3d77c86dbc976d3859f2fca045df553517650950da6d46c0513"} Nov 03 20:23:25.533838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:25.533786 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" podStartSLOduration=1.5337678270000001 podStartE2EDuration="1.533767827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:25.528275283 +0000 UTC m=+1583.903872033" watchObservedRunningTime="2026-11-03 20:23:25.533767827 +0000 UTC m=+1583.909364576" Nov 03 20:23:25.553621 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:25.553575 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" podStartSLOduration=1.553561939 podStartE2EDuration="1.553561939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:25.549798813 +0000 UTC m=+1583.925395562" watchObservedRunningTime="2026-11-03 20:23:25.553561939 +0000 UTC m=+1583.929158686" Nov 03 20:23:27.773326 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:27.773282 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q"] Nov 03 20:23:27.806171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:27.805478 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:27.816380 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:27.812610 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q"] Nov 03 20:23:27.816380 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:27.815194 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d9f082-openshift-default-dockercfg-mggr7\"" Nov 03 20:23:28.001407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:27.986434 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-credential-socket\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.001407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:27.986578 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-socket\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.001407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:27.986648 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istiod-ca-cert\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.001407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:27.986705 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnqpt\" (UniqueName: \"kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-kube-api-access-dnqpt\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.001407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:27.986747 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-token\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.001407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:27.986821 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-certs\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.001407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:27.986898 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-envoy\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.001407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:27.986956 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-podinfo\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.001407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:27.987056 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-data\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.097410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.096484 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istiod-ca-cert\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.097410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.096569 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnqpt\" (UniqueName: \"kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-kube-api-access-dnqpt\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.097410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.096617 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-token\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.097410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.096692 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-certs\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.097410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.096774 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-envoy\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.097410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.096835 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-podinfo\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.097410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.096931 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-data\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.097410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.096995 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-credential-socket\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.097410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.097108 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-socket\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.098037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.097632 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-socket\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.106682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.098273 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istiod-ca-cert\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.106682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.099040 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-certs\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.106682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.099648 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-data\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.131856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.131813 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-credential-socket\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.151411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.141209 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-envoy\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.151411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.141821 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-podinfo\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.151411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.144119 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-token\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.176635 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.152097 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnqpt\" (UniqueName: \"kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-kube-api-access-dnqpt\") pod \"e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.429224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.429136 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:28.830332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.830294 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q"] Nov 03 20:23:28.835616 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.835568 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:28.835874 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.835812 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:28.835995 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:28.835899 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:29.614738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:29.614691 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" event={"ID":"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2","Type":"ContainerStarted","Data":"372a03fdf78fdf92ee8b8db1f9918d1ac307245577fe1431ea379afc88676a45"} Nov 03 20:23:29.614961 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:29.614808 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" event={"ID":"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2","Type":"ContainerStarted","Data":"5d5d65fa3dd5d21620af012e47ac1660e5a150a620d02fbd9b7833a952be41d9"} Nov 03 20:23:29.643945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:29.643891 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" podStartSLOduration=2.643876412 podStartE2EDuration="2.643876412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:29.638960315 +0000 UTC m=+1588.014557063" watchObservedRunningTime="2026-11-03 20:23:29.643876412 +0000 UTC m=+1588.019473218" Nov 03 20:23:30.430251 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:30.430213 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:30.437576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:30.437549 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:30.623979 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:30.623951 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:30.624891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:30.624873 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:23:33.704093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:33.704056 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f"] Nov 03 20:23:33.709359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:33.709335 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" Nov 03 20:23:33.723870 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:33.719124 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d9f082\"" Nov 03 20:23:33.723870 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:33.719207 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d9f082-dockercfg-mlzvj\"" Nov 03 20:23:33.725232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:33.725207 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f"] Nov 03 20:23:34.024322 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:34.023834 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99xq2\" (UniqueName: \"kubernetes.io/projected/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-kube-api-access-99xq2\") pod \"payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f\" (UID: \"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" Nov 03 20:23:34.024917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:34.024870 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f\" (UID: \"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" Nov 03 20:23:34.126306 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:34.126263 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99xq2\" (UniqueName: \"kubernetes.io/projected/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-kube-api-access-99xq2\") pod \"payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f\" (UID: \"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" Nov 03 20:23:34.134561 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:34.134520 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f\" (UID: \"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" Nov 03 20:23:34.190597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:34.136074 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f\" (UID: \"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" Nov 03 20:23:34.206783 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:34.206728 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99xq2\" (UniqueName: \"kubernetes.io/projected/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-kube-api-access-99xq2\") pod \"payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f\" (UID: \"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" Nov 03 20:23:34.330977 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:34.330929 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" Nov 03 20:23:35.063340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:35.063266 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f"] Nov 03 20:23:35.089881 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:23:35.089838 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod037b282c_3c0b_4fa5_bcdc_eb0f4aa1babf.slice/crio-48a5e6e6797cf534f399e04d5a0b36562a046f2c9bc9e8fe1e5d75732c562212 WatchSource:0}: Error finding container 48a5e6e6797cf534f399e04d5a0b36562a046f2c9bc9e8fe1e5d75732c562212: Status 404 returned error can't find the container with id 48a5e6e6797cf534f399e04d5a0b36562a046f2c9bc9e8fe1e5d75732c562212 Nov 03 20:23:35.729771 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:35.729734 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" event={"ID":"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf","Type":"ContainerStarted","Data":"9c4bb3151f01743d711ba215e08f99cb373b82f857ed12d7ebbbfb94707d7aa5"} Nov 03 20:23:35.729948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:35.729795 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" event={"ID":"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf","Type":"ContainerStarted","Data":"48a5e6e6797cf534f399e04d5a0b36562a046f2c9bc9e8fe1e5d75732c562212"} Nov 03 20:23:35.729948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:35.729828 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" Nov 03 20:23:35.751694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:35.751637 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" podStartSLOduration=2.751623759 podStartE2EDuration="2.751623759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:35.747922884 +0000 UTC m=+1594.123519633" watchObservedRunningTime="2026-11-03 20:23:35.751623759 +0000 UTC m=+1594.127220505" Nov 03 20:23:36.107214 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.107180 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl"] Nov 03 20:23:36.112279 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.112256 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" Nov 03 20:23:36.117126 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.117099 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f3ae23\"" Nov 03 20:23:36.123031 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.122997 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl"] Nov 03 20:23:36.149434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.149353 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs"] Nov 03 20:23:36.155434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.155356 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" Nov 03 20:23:36.162341 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.162315 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f3ae23-dockercfg-vnndd\"" Nov 03 20:23:36.170930 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.170904 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs"] Nov 03 20:23:36.185956 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:23:36.185917 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:23:36.217585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.217544 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcvxd\" (UniqueName: \"kubernetes.io/projected/14547b1f-02fe-43a7-b6c1-c2165dac0f17-kube-api-access-gcvxd\") pod \"payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl\" (UID: \"14547b1f-02fe-43a7-b6c1-c2165dac0f17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" Nov 03 20:23:36.217899 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.217873 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14547b1f-02fe-43a7-b6c1-c2165dac0f17-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl\" (UID: \"14547b1f-02fe-43a7-b6c1-c2165dac0f17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" Nov 03 20:23:36.323037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.322992 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14547b1f-02fe-43a7-b6c1-c2165dac0f17-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl\" (UID: \"14547b1f-02fe-43a7-b6c1-c2165dac0f17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" Nov 03 20:23:36.323249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.323176 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bfef0b-3309-464e-97ed-2eec4aa76d9b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs\" (UID: \"73bfef0b-3309-464e-97ed-2eec4aa76d9b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" Nov 03 20:23:36.323249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.323232 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t587\" (UniqueName: \"kubernetes.io/projected/73bfef0b-3309-464e-97ed-2eec4aa76d9b-kube-api-access-5t587\") pod \"payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs\" (UID: \"73bfef0b-3309-464e-97ed-2eec4aa76d9b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" Nov 03 20:23:36.323378 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.323327 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcvxd\" (UniqueName: \"kubernetes.io/projected/14547b1f-02fe-43a7-b6c1-c2165dac0f17-kube-api-access-gcvxd\") pod \"payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl\" (UID: \"14547b1f-02fe-43a7-b6c1-c2165dac0f17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" Nov 03 20:23:36.347459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.347421 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14547b1f-02fe-43a7-b6c1-c2165dac0f17-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl\" (UID: \"14547b1f-02fe-43a7-b6c1-c2165dac0f17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" Nov 03 20:23:36.354467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.351351 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcvxd\" (UniqueName: \"kubernetes.io/projected/14547b1f-02fe-43a7-b6c1-c2165dac0f17-kube-api-access-gcvxd\") pod \"payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl\" (UID: \"14547b1f-02fe-43a7-b6c1-c2165dac0f17\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" Nov 03 20:23:36.430458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.428091 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bfef0b-3309-464e-97ed-2eec4aa76d9b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs\" (UID: \"73bfef0b-3309-464e-97ed-2eec4aa76d9b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" Nov 03 20:23:36.430458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.428163 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t587\" (UniqueName: \"kubernetes.io/projected/73bfef0b-3309-464e-97ed-2eec4aa76d9b-kube-api-access-5t587\") pod \"payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs\" (UID: \"73bfef0b-3309-464e-97ed-2eec4aa76d9b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" Nov 03 20:23:36.430458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.429876 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bfef0b-3309-464e-97ed-2eec4aa76d9b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs\" (UID: \"73bfef0b-3309-464e-97ed-2eec4aa76d9b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" Nov 03 20:23:36.456413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.450939 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" Nov 03 20:23:36.473254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.473225 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t587\" (UniqueName: \"kubernetes.io/projected/73bfef0b-3309-464e-97ed-2eec4aa76d9b-kube-api-access-5t587\") pod \"payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs\" (UID: \"73bfef0b-3309-464e-97ed-2eec4aa76d9b\") " pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" Nov 03 20:23:36.501909 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.501870 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" Nov 03 20:23:36.538668 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.538136 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" Nov 03 20:23:36.540424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.540006 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" Nov 03 20:23:36.852779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.848050 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl"] Nov 03 20:23:36.901951 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:36.901903 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs"] Nov 03 20:23:36.903980 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:23:36.903907 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73bfef0b_3309_464e_97ed_2eec4aa76d9b.slice/crio-af95295d144fef92112fc73dae8e70bc61bf58bc3d10eccd3886a46df87d34d5 WatchSource:0}: Error finding container af95295d144fef92112fc73dae8e70bc61bf58bc3d10eccd3886a46df87d34d5: Status 404 returned error can't find the container with id af95295d144fef92112fc73dae8e70bc61bf58bc3d10eccd3886a46df87d34d5 Nov 03 20:23:37.784187 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:37.784135 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" event={"ID":"73bfef0b-3309-464e-97ed-2eec4aa76d9b","Type":"ContainerStarted","Data":"b5936e827bb7f6d742901718e5a55c88c45b1b370ae13c5975c7faeb7333d30b"} Nov 03 20:23:37.784705 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:37.784208 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" event={"ID":"73bfef0b-3309-464e-97ed-2eec4aa76d9b","Type":"ContainerStarted","Data":"af95295d144fef92112fc73dae8e70bc61bf58bc3d10eccd3886a46df87d34d5"} Nov 03 20:23:37.785536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:37.785510 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" Nov 03 20:23:37.787833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:37.787793 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" event={"ID":"14547b1f-02fe-43a7-b6c1-c2165dac0f17","Type":"ContainerStarted","Data":"8492a32a4faf34988dbac18f4358548536b195483a4fa8f2c85d7cc014c70297"} Nov 03 20:23:37.787986 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:37.787856 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" event={"ID":"14547b1f-02fe-43a7-b6c1-c2165dac0f17","Type":"ContainerStarted","Data":"bc9d36c401dacb2f06c1d35fc9787aab378108f629c720bafb838f401e04396f"} Nov 03 20:23:37.787986 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:37.787915 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" Nov 03 20:23:37.806962 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:37.806904 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" podStartSLOduration=1.806892281 podStartE2EDuration="1.806892281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:37.803697329 +0000 UTC m=+1596.179294076" watchObservedRunningTime="2026-11-03 20:23:37.806892281 +0000 UTC m=+1596.182489027" Nov 03 20:23:37.848168 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:37.848117 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" podStartSLOduration=1.848104229 podStartE2EDuration="1.848104229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:37.842888844 +0000 UTC m=+1596.218485597" watchObservedRunningTime="2026-11-03 20:23:37.848104229 +0000 UTC m=+1596.223700975" Nov 03 20:23:42.680498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:42.680453 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f"] Nov 03 20:23:42.714215 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:42.714173 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e50c04-jbn8f"] Nov 03 20:23:43.766676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:43.766639 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b0f7519-da9f-4e59-a8ae-64d16862eef2" path="/var/lib/kubelet/pods/1b0f7519-da9f-4e59-a8ae-64d16862eef2/volumes" Nov 03 20:23:46.764877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:46.764842 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" Nov 03 20:23:48.298653 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.298618 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk"] Nov 03 20:23:48.305819 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.305789 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.312297 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.311938 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk"] Nov 03 20:23:48.350322 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.350286 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk"] Nov 03 20:23:48.350682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.350653 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.387907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.383638 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk"] Nov 03 20:23:48.387907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.386211 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hffvt\" (UniqueName: \"kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-kube-api-access-hffvt\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.387907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.386267 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.387907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.386327 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.386379 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.390196 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.390294 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6454d0c8-5104-4554-ab8d-4f822b372bb2-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.390379 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.390449 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.390505 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.390549 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.390587 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.390729 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm5jw\" (UniqueName: \"kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-kube-api-access-fm5jw\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.390785 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.390832 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.390861 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.390892 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.391026 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.403143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.391060 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.492638 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.492596 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.492850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.492666 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.492850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.492715 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.492850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.492761 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6454d0c8-5104-4554-ab8d-4f822b372bb2-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.492850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.492812 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.493161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.492855 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.493161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.492901 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.493161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.492934 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.493161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.492962 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.493161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.493081 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm5jw\" (UniqueName: \"kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-kube-api-access-fm5jw\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.493161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.493123 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.493161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.493159 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.493568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.493183 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.493568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.493210 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.493568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.493322 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.493568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.493346 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.493568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.493422 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hffvt\" (UniqueName: \"kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-kube-api-access-hffvt\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.493568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.493465 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.493568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.493490 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.493943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.493604 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.493943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.493763 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.493943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.493830 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-credential-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.494106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.494080 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.494582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.494555 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-data\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.494853 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.494829 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-socket\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.495006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.494977 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6454d0c8-5104-4554-ab8d-4f822b372bb2-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.495181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.495160 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-certs\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.495331 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.495310 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istiod-ca-cert\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.495813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.495789 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.496568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.496545 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.497170 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.497149 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-envoy\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.498892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.498869 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-podinfo\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.505292 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.505265 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.505840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.505813 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-token\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.506224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.506206 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm5jw\" (UniqueName: \"kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-kube-api-access-fm5jw\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.506585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.506560 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hffvt\" (UniqueName: \"kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-kube-api-access-hffvt\") pod \"e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.671438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.671306 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:48.681644 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.681620 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:48.803441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:48.802721 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" Nov 03 20:23:49.011228 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:49.010122 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk"] Nov 03 20:23:49.024624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:49.024578 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:49.024764 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:49.024666 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk"] Nov 03 20:23:49.024827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:49.024672 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:49.024891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:49.024863 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:49.025097 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:49.025071 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:49.025167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:49.025148 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:49.025223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:49.025200 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:23:49.816864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:49.816832 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" Nov 03 20:23:50.182466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.178619 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d"] Nov 03 20:23:50.186409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.185877 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ccdee44-vgj4d"] Nov 03 20:23:50.240332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.239638 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" event={"ID":"cc55dea0-c60e-43f3-b4d1-43b0f9486226","Type":"ContainerStarted","Data":"2725a8fa0ffacb6e576c76004e1aa5914df8fab49d1f342e605055fb8172f7de"} Nov 03 20:23:50.240332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.239705 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" event={"ID":"cc55dea0-c60e-43f3-b4d1-43b0f9486226","Type":"ContainerStarted","Data":"ba1b8373eb503fce0e8972566d7abe3d0b72e1fb979da6d5045d279e30d8da23"} Nov 03 20:23:50.243197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.243160 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" event={"ID":"6454d0c8-5104-4554-ab8d-4f822b372bb2","Type":"ContainerStarted","Data":"e2f0e5ab4cf200747edd792160556fc2e662b7d0e6edf81652ac8bcf900882fe"} Nov 03 20:23:50.243356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.243222 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" event={"ID":"6454d0c8-5104-4554-ab8d-4f822b372bb2","Type":"ContainerStarted","Data":"0757e75011973159503a4c38aa50cc38b42677688141cdda6979d7482d7e6094"} Nov 03 20:23:50.323648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.323583 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" podStartSLOduration=2.323561426 podStartE2EDuration="2.323561426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:50.263104189 +0000 UTC m=+1608.638700948" watchObservedRunningTime="2026-11-03 20:23:50.323561426 +0000 UTC m=+1608.699158176" Nov 03 20:23:50.343701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.343642 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" podStartSLOduration=2.343623112 podStartE2EDuration="2.343623112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:23:50.287911163 +0000 UTC m=+1608.663507908" watchObservedRunningTime="2026-11-03 20:23:50.343623112 +0000 UTC m=+1608.719219861" Nov 03 20:23:50.682411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.678545 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:50.682653 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.682525 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:50.690419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.689543 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:23:50.690419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.689614 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" podUID="6454d0c8-5104-4554-ab8d-4f822b372bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 20:23:50.690419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.689997 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:23:50.690419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:50.690036 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 20:23:51.677408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:51.677354 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:23:51.677926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:51.677438 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 20:23:51.693803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:51.693766 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:23:51.694080 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:51.694048 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" podUID="6454d0c8-5104-4554-ab8d-4f822b372bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 20:23:51.790858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:51.790819 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96ddaa2d-106d-4537-8421-130805474905" path="/var/lib/kubelet/pods/96ddaa2d-106d-4537-8421-130805474905/volumes" Nov 03 20:23:52.673911 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:52.673766 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:23:52.673911 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:52.673843 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 20:23:52.734428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:52.734375 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:23:52.748596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:52.738474 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" podUID="6454d0c8-5104-4554-ab8d-4f822b372bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 20:23:52.864678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:52.864615 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:52.942970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:52.942871 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:52.988410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:52.983471 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:53.018481 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.018438 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:53.062448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.058227 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:53.113531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.099000 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:53.154065 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.148282 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:53.329414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.322638 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:53.421641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.421578 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:53.507554 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.507496 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Nov 03 20:23:53.539053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.539008 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:53.697548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.693544 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:23:53.697548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.693621 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 20:23:53.698138 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.698030 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:23:53.698138 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.698082 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" podUID="6454d0c8-5104-4554-ab8d-4f822b372bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 20:23:53.763486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.763444 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:23:53.788532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.768471 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:23:53.929521 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.929474 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:53.930298 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.930270 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:23:53.950440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.950296 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:53.951279 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:53.951257 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:23:54.126511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:54.126455 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Nov 03 20:23:54.142406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:54.127778 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:23:54.144856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:54.143264 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:23:54.172271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:54.172225 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:54.189076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:54.189042 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:23:54.427587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:54.427544 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:23:54.674491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:54.674455 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:23:54.697833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:54.697728 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 20:23:54.698083 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:54.697064 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:23:54.698234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:54.698210 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" podUID="6454d0c8-5104-4554-ab8d-4f822b372bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 20:23:55.026598 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.026482 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:55.133412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.129470 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:23:55.133412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.130552 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:55.133412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.131055 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Nov 03 20:23:55.145414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.144648 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:55.182408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.181473 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:55.182625 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.182491 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:55.186408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.182819 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:55.186408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.182880 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:23:55.186408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.183513 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:23:55.186408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.183801 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:23:55.262450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.258508 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded" Nov 03 20:23:55.262450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.258789 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 03 20:23:55.262450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.260210 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 03 20:23:55.427411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.406615 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:55.429784 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.429738 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:23:55.519415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.517067 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:23:55.535553 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.535508 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": context deadline exceeded" Nov 03 20:23:55.681144 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.681046 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:23:55.681322 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.681131 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 20:23:55.698266 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.698231 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:23:55.698557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.698528 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" podUID="6454d0c8-5104-4554-ab8d-4f822b372bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 20:23:55.698725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.698706 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:55.768525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.768479 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:23:55.774780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.774740 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:23:55.872672 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:55.872629 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded" Nov 03 20:23:56.028570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.028469 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:56.134521 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.134468 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:56.135049 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.135024 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:23:56.146559 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.146518 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:56.148354 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.148314 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:23:56.150621 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.150579 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:56.253838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.253770 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:56.277468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.263037 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:56.277468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.264120 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:23:56.277468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.264529 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:56.277468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.264959 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:23:56.433979 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.431226 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:23:56.433979 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.431816 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:23:56.556428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.536547 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:56.556428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.537071 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:23:56.672303 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.670531 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:23:56.674747 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.674639 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:23:56.674747 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.674709 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 20:23:56.714502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.694987 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:23:56.714502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.695040 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" podUID="6454d0c8-5104-4554-ab8d-4f822b372bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 20:23:56.777558 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.777511 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:23:56.811423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.809543 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:23:56.843696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.843666 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:23:56.843928 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.843917 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:23:56.902466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:56.898820 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:23:57.005462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:57.004013 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:23:57.055539 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:57.040883 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:57.190418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:57.186844 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:23:57.195621 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:57.194956 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:57.467195 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:57.467142 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 03 20:23:57.469520 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:57.469455 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:57.501103 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:57.500178 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:23:57.677421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:57.674729 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:23:57.677421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:57.674806 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 20:23:57.689419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:57.686046 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Nov 03 20:23:57.689419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:57.686101 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" podUID="6454d0c8-5104-4554-ab8d-4f822b372bb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Nov 03 20:23:57.747582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:57.747493 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:23:58.206421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:58.205499 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:23:58.412187 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:58.412155 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:23:58.604700 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:58.604670 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:23:58.732797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:58.732763 4457 patch_prober.go:29] interesting pod/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 03 20:23:58.733113 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:58.733084 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 03 20:23:58.827841 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:58.827481 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:58.829917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:58.829891 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:58.833521 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:58.833497 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:23:59.677772 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:59.677740 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:59.678608 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:59.678518 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:23:59.679550 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:23:59.679530 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:24:04.905003 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:04.904955 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk"] Nov 03 20:24:04.905876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:04.905808 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerName="istio-proxy" containerID="cri-o://2725a8fa0ffacb6e576c76004e1aa5914df8fab49d1f342e605055fb8172f7de" gracePeriod=30 Nov 03 20:24:04.972442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:04.972410 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk"] Nov 03 20:24:04.973212 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:04.973175 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" podUID="6454d0c8-5104-4554-ab8d-4f822b372bb2" containerName="istio-proxy" containerID="cri-o://e2f0e5ab4cf200747edd792160556fc2e662b7d0e6edf81652ac8bcf900882fe" gracePeriod=30 Nov 03 20:24:05.100349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:05.100296 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b"] Nov 03 20:24:05.101273 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:05.101003 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" podUID="1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" containerName="istio-proxy" containerID="cri-o://4b71cf7c808ac46bbd87a5430dc33ab701f36c8fec9fe99c87c52307fc3eddb7" gracePeriod=30 Nov 03 20:24:06.152311 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:24:06.152264 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:24:06.152787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.152763 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9"] Nov 03 20:24:06.160146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.160121 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" Nov 03 20:24:06.167717 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.167686 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7ae0f1-dockercfg-8tnnw\"" Nov 03 20:24:06.167864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.167731 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7ae0f1\"" Nov 03 20:24:06.170530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.170506 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5"] Nov 03 20:24:06.179955 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.179913 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" Nov 03 20:24:06.181765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.181688 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9"] Nov 03 20:24:06.192554 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.191924 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5"] Nov 03 20:24:06.233029 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.233003 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9dm\" (UniqueName: \"kubernetes.io/projected/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-kube-api-access-qp9dm\") pod \"payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9\" (UID: \"fad0ed4c-3ba5-4a1c-a221-668576f5d27e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" Nov 03 20:24:06.233176 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.233134 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9\" (UID: \"fad0ed4c-3ba5-4a1c-a221-668576f5d27e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" Nov 03 20:24:06.233234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.233215 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5lqv\" (UniqueName: \"kubernetes.io/projected/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-kube-api-access-l5lqv\") pod \"payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5\" (UID: \"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" Nov 03 20:24:06.233366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.233354 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5\" (UID: \"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" Nov 03 20:24:06.339059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.334500 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5\" (UID: \"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" Nov 03 20:24:06.339059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.334689 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9dm\" (UniqueName: \"kubernetes.io/projected/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-kube-api-access-qp9dm\") pod \"payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9\" (UID: \"fad0ed4c-3ba5-4a1c-a221-668576f5d27e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" Nov 03 20:24:06.339059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.334809 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9\" (UID: \"fad0ed4c-3ba5-4a1c-a221-668576f5d27e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" Nov 03 20:24:06.344416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.334860 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5lqv\" (UniqueName: \"kubernetes.io/projected/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-kube-api-access-l5lqv\") pod \"payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5\" (UID: \"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" Nov 03 20:24:06.344416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.343924 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5\" (UID: \"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" Nov 03 20:24:06.347478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.345095 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9\" (UID: \"fad0ed4c-3ba5-4a1c-a221-668576f5d27e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" Nov 03 20:24:06.406462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.402359 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5lqv\" (UniqueName: \"kubernetes.io/projected/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-kube-api-access-l5lqv\") pod \"payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5\" (UID: \"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" Nov 03 20:24:06.406462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.405062 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9dm\" (UniqueName: \"kubernetes.io/projected/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-kube-api-access-qp9dm\") pod \"payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9\" (UID: \"fad0ed4c-3ba5-4a1c-a221-668576f5d27e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" Nov 03 20:24:06.498461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.495014 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" Nov 03 20:24:06.516959 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.516684 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" Nov 03 20:24:06.574377 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.574339 4457 scope.go:122] "RemoveContainer" containerID="54474491632b60ee9dad98d39fdb72cdb4661fd9e87e865f3060eff7f5c71abd" Nov 03 20:24:06.611798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.610374 4457 scope.go:122] "RemoveContainer" containerID="e697cf117650a9f3fb8f42f1687eba15e51ddd784f6192bf580495359de7b37b" Nov 03 20:24:06.863903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.863865 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5"] Nov 03 20:24:06.903874 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:24:06.901930 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad0ed4c_3ba5_4a1c_a221_668576f5d27e.slice/crio-a017406fd4f9f8997f1d34c7bea19bc3146b6c32622f61e10d9241fb1379aec8 WatchSource:0}: Error finding container a017406fd4f9f8997f1d34c7bea19bc3146b6c32622f61e10d9241fb1379aec8: Status 404 returned error can't find the container with id a017406fd4f9f8997f1d34c7bea19bc3146b6c32622f61e10d9241fb1379aec8 Nov 03 20:24:06.904116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:06.903981 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9"] Nov 03 20:24:07.476811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:07.476761 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" event={"ID":"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7","Type":"ContainerStarted","Data":"4d051ef325f777b6f42b652fc23c61da1dadc49e790c42debe1211bed9e1700f"} Nov 03 20:24:07.477280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:07.476828 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" event={"ID":"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7","Type":"ContainerStarted","Data":"1dda7eb4726066b7b2db15a8f7089e7ffaf5d59dfc53c018384477eb5d2817fb"} Nov 03 20:24:07.477280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:07.477024 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" Nov 03 20:24:07.478895 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:07.478871 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" event={"ID":"fad0ed4c-3ba5-4a1c-a221-668576f5d27e","Type":"ContainerStarted","Data":"91d322d9a065425bc4e4284396cc9c8878aa58ac59c9e4ce80bd6a27ed5503d5"} Nov 03 20:24:07.479033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:07.478913 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" event={"ID":"fad0ed4c-3ba5-4a1c-a221-668576f5d27e","Type":"ContainerStarted","Data":"a017406fd4f9f8997f1d34c7bea19bc3146b6c32622f61e10d9241fb1379aec8"} Nov 03 20:24:07.479033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:07.479005 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" Nov 03 20:24:07.508581 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:07.508515 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" podStartSLOduration=1.508498162 podStartE2EDuration="1.508498162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:07.497430469 +0000 UTC m=+1625.873027218" watchObservedRunningTime="2026-11-03 20:24:07.508498162 +0000 UTC m=+1625.884094909" Nov 03 20:24:07.535701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:07.535642 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" podStartSLOduration=1.535623164 podStartE2EDuration="1.535623164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:07.532913172 +0000 UTC m=+1625.908509931" watchObservedRunningTime="2026-11-03 20:24:07.535623164 +0000 UTC m=+1625.911219912" Nov 03 20:24:10.308421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.307740 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:24:10.425333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.425293 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-credential-socket\") pod \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " Nov 03 20:24:10.425523 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.425358 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istiod-ca-cert\") pod \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " Nov 03 20:24:10.425523 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.425493 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-envoy\") pod \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " Nov 03 20:24:10.425681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.425659 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-socket\") pod \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " Nov 03 20:24:10.425743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.425700 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-podinfo\") pod \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " Nov 03 20:24:10.425842 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.425825 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-token\" (UniqueName: \"kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-token\") pod \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " Nov 03 20:24:10.425931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.425916 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-kube-api-access-hffvt\" (UniqueName: \"kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-kube-api-access-hffvt\") pod \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " Nov 03 20:24:10.425993 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.425970 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-data\") pod \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " Nov 03 20:24:10.426051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.426017 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-certs\") pod \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\" (UID: \"cc55dea0-c60e-43f3-b4d1-43b0f9486226\") " Nov 03 20:24:10.426206 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.426163 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istiod-ca-cert" pod "cc55dea0-c60e-43f3-b4d1-43b0f9486226" (UID: "cc55dea0-c60e-43f3-b4d1-43b0f9486226"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:10.426714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.426686 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-credential-socket" pod "cc55dea0-c60e-43f3-b4d1-43b0f9486226" (UID: "cc55dea0-c60e-43f3-b4d1-43b0f9486226"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.427034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.426846 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-certs" pod "cc55dea0-c60e-43f3-b4d1-43b0f9486226" (UID: "cc55dea0-c60e-43f3-b4d1-43b0f9486226"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.427034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.427028 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.427198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.427042 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-data" pod "cc55dea0-c60e-43f3-b4d1-43b0f9486226" (UID: "cc55dea0-c60e-43f3-b4d1-43b0f9486226"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.427198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.427047 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.427198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.427081 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.427374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.427222 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-socket" pod "cc55dea0-c60e-43f3-b4d1-43b0f9486226" (UID: "cc55dea0-c60e-43f3-b4d1-43b0f9486226"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.429594 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.429568 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-token" pod "cc55dea0-c60e-43f3-b4d1-43b0f9486226" (UID: "cc55dea0-c60e-43f3-b4d1-43b0f9486226"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.429870 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.429845 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-envoy" pod "cc55dea0-c60e-43f3-b4d1-43b0f9486226" (UID: "cc55dea0-c60e-43f3-b4d1-43b0f9486226"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.430467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.430443 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-kube-api-access-hffvt" pod "cc55dea0-c60e-43f3-b4d1-43b0f9486226" (UID: "cc55dea0-c60e-43f3-b4d1-43b0f9486226"). InnerVolumeSpecName "kube-api-access-hffvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.430668 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.430637 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-podinfo" pod "cc55dea0-c60e-43f3-b4d1-43b0f9486226" (UID: "cc55dea0-c60e-43f3-b4d1-43b0f9486226"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:10.511592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.497335 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:24:10.528454 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.528302 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-data\") pod \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " Nov 03 20:24:10.528454 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.528369 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-kube-api-access-5qlzm\" (UniqueName: \"kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-kube-api-access-5qlzm\") pod \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " Nov 03 20:24:10.529329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.528986 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-data" pod "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" (UID: "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.529329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.529026 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istiod-ca-cert\") pod \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " Nov 03 20:24:10.529329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.529210 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-podinfo\") pod \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " Nov 03 20:24:10.529329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.529253 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-token\" (UniqueName: \"kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-token\") pod \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " Nov 03 20:24:10.529329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.529304 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-envoy\") pod \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " Nov 03 20:24:10.529689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.529428 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-credential-socket\") pod \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " Nov 03 20:24:10.529689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.529465 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-socket\") pod \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " Nov 03 20:24:10.529689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.529557 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-certs\") pod \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\" (UID: \"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1\") " Nov 03 20:24:10.530699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.530380 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-credential-socket" pod "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" (UID: "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.530699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.530420 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.530699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.530443 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.530699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.530458 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.530699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.530488 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hffvt\" (UniqueName: \"kubernetes.io/projected/cc55dea0-c60e-43f3-b4d1-43b0f9486226-kube-api-access-hffvt\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.530699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.530505 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.530699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.530519 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.530699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.530533 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc55dea0-c60e-43f3-b4d1-43b0f9486226-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.531188 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.530710 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-certs" pod "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" (UID: "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.531188 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.530879 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-socket" pod "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" (UID: "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.531188 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.531158 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istiod-ca-cert" pod "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" (UID: "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:10.545522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.543785 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-token" pod "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" (UID: "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.545522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.543902 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-kube-api-access-5qlzm" pod "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" (UID: "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1"). InnerVolumeSpecName "kube-api-access-5qlzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.545717 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.545696 4457 generic.go:356] "Generic (PLEG): container finished" podID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerID="2725a8fa0ffacb6e576c76004e1aa5914df8fab49d1f342e605055fb8172f7de" exitCode=0 Nov 03 20:24:10.545838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.545801 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" event={"ID":"cc55dea0-c60e-43f3-b4d1-43b0f9486226","Type":"ContainerDied","Data":"2725a8fa0ffacb6e576c76004e1aa5914df8fab49d1f342e605055fb8172f7de"} Nov 03 20:24:10.545838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.545825 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-podinfo" pod "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" (UID: "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:10.545968 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.545903 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" event={"ID":"cc55dea0-c60e-43f3-b4d1-43b0f9486226","Type":"ContainerDied","Data":"ba1b8373eb503fce0e8972566d7abe3d0b72e1fb979da6d5045d279e30d8da23"} Nov 03 20:24:10.545968 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.545926 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-envoy" pod "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" (UID: "1f5ce6fb-4dca-4cd7-befa-82ea92a023b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.545968 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.545936 4457 scope.go:122] "RemoveContainer" containerID="2725a8fa0ffacb6e576c76004e1aa5914df8fab49d1f342e605055fb8172f7de" Nov 03 20:24:10.546131 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.546026 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk" Nov 03 20:24:10.553201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.552985 4457 generic.go:356] "Generic (PLEG): container finished" podID="6454d0c8-5104-4554-ab8d-4f822b372bb2" containerID="e2f0e5ab4cf200747edd792160556fc2e662b7d0e6edf81652ac8bcf900882fe" exitCode=0 Nov 03 20:24:10.553201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.553117 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" event={"ID":"6454d0c8-5104-4554-ab8d-4f822b372bb2","Type":"ContainerDied","Data":"e2f0e5ab4cf200747edd792160556fc2e662b7d0e6edf81652ac8bcf900882fe"} Nov 03 20:24:10.558280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.556483 4457 generic.go:356] "Generic (PLEG): container finished" podID="1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" containerID="4b71cf7c808ac46bbd87a5430dc33ab701f36c8fec9fe99c87c52307fc3eddb7" exitCode=0 Nov 03 20:24:10.558280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.556669 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" event={"ID":"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1","Type":"ContainerDied","Data":"4b71cf7c808ac46bbd87a5430dc33ab701f36c8fec9fe99c87c52307fc3eddb7"} Nov 03 20:24:10.558280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.556709 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" event={"ID":"1f5ce6fb-4dca-4cd7-befa-82ea92a023b1","Type":"ContainerDied","Data":"aac3f6f54b0f7f7354a60da70f6cfd927bc2dc27506d53c89531c31154c42a00"} Nov 03 20:24:10.558280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.556784 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b" Nov 03 20:24:10.559010 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.558989 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:24:10.567970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.567952 4457 scope.go:122] "RemoveContainer" containerID="2725a8fa0ffacb6e576c76004e1aa5914df8fab49d1f342e605055fb8172f7de" Nov 03 20:24:10.568592 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:24:10.568465 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2725a8fa0ffacb6e576c76004e1aa5914df8fab49d1f342e605055fb8172f7de\": container with ID starting with 2725a8fa0ffacb6e576c76004e1aa5914df8fab49d1f342e605055fb8172f7de not found: ID does not exist" containerID="2725a8fa0ffacb6e576c76004e1aa5914df8fab49d1f342e605055fb8172f7de" Nov 03 20:24:10.568592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.568497 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2725a8fa0ffacb6e576c76004e1aa5914df8fab49d1f342e605055fb8172f7de"} err="failed to get container status \"2725a8fa0ffacb6e576c76004e1aa5914df8fab49d1f342e605055fb8172f7de\": rpc error: code = NotFound desc = could not find container \"2725a8fa0ffacb6e576c76004e1aa5914df8fab49d1f342e605055fb8172f7de\": container with ID starting with 2725a8fa0ffacb6e576c76004e1aa5914df8fab49d1f342e605055fb8172f7de not found: ID does not exist" Nov 03 20:24:10.568592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.568522 4457 scope.go:122] "RemoveContainer" containerID="4b71cf7c808ac46bbd87a5430dc33ab701f36c8fec9fe99c87c52307fc3eddb7" Nov 03 20:24:10.627200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.627174 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk"] Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.632192 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6454d0c8-5104-4554-ab8d-4f822b372bb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6454d0c8-5104-4554-ab8d-4f822b372bb2-istiod-ca-cert\") pod \"6454d0c8-5104-4554-ab8d-4f822b372bb2\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.632237 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-kube-api-access-fm5jw\" (UniqueName: \"kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-kube-api-access-fm5jw\") pod \"6454d0c8-5104-4554-ab8d-4f822b372bb2\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.632322 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-certs\") pod \"6454d0c8-5104-4554-ab8d-4f822b372bb2\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.632419 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-podinfo\") pod \"6454d0c8-5104-4554-ab8d-4f822b372bb2\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.632445 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-envoy\") pod \"6454d0c8-5104-4554-ab8d-4f822b372bb2\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.632490 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-data\") pod \"6454d0c8-5104-4554-ab8d-4f822b372bb2\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.632534 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-credential-socket\") pod \"6454d0c8-5104-4554-ab8d-4f822b372bb2\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.632637 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-token\" (UniqueName: \"kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-token\") pod \"6454d0c8-5104-4554-ab8d-4f822b372bb2\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.632665 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-socket\") pod \"6454d0c8-5104-4554-ab8d-4f822b372bb2\" (UID: \"6454d0c8-5104-4554-ab8d-4f822b372bb2\") " Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.633323 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.633342 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.633356 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.633372 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.633403 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.633418 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.633433 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qlzm\" (UniqueName: \"kubernetes.io/projected/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-kube-api-access-5qlzm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.636742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.633448 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.640736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.639731 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-socket" pod "6454d0c8-5104-4554-ab8d-4f822b372bb2" (UID: "6454d0c8-5104-4554-ab8d-4f822b372bb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.641282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.640994 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6454d0c8-5104-4554-ab8d-4f822b372bb2-istiod-ca-cert" pod "6454d0c8-5104-4554-ab8d-4f822b372bb2" (UID: "6454d0c8-5104-4554-ab8d-4f822b372bb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:10.641812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.641695 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-29bhk"] Nov 03 20:24:10.648236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.643738 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-certs" pod "6454d0c8-5104-4554-ab8d-4f822b372bb2" (UID: "6454d0c8-5104-4554-ab8d-4f822b372bb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.648236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.644258 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-credential-socket" pod "6454d0c8-5104-4554-ab8d-4f822b372bb2" (UID: "6454d0c8-5104-4554-ab8d-4f822b372bb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.648236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.644441 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-data" pod "6454d0c8-5104-4554-ab8d-4f822b372bb2" (UID: "6454d0c8-5104-4554-ab8d-4f822b372bb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.648236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.647951 4457 scope.go:122] "RemoveContainer" containerID="4b71cf7c808ac46bbd87a5430dc33ab701f36c8fec9fe99c87c52307fc3eddb7" Nov 03 20:24:10.648892 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:24:10.648757 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b71cf7c808ac46bbd87a5430dc33ab701f36c8fec9fe99c87c52307fc3eddb7\": container with ID starting with 4b71cf7c808ac46bbd87a5430dc33ab701f36c8fec9fe99c87c52307fc3eddb7 not found: ID does not exist" containerID="4b71cf7c808ac46bbd87a5430dc33ab701f36c8fec9fe99c87c52307fc3eddb7" Nov 03 20:24:10.648892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.648793 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-podinfo" pod "6454d0c8-5104-4554-ab8d-4f822b372bb2" (UID: "6454d0c8-5104-4554-ab8d-4f822b372bb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:10.648892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.648798 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b71cf7c808ac46bbd87a5430dc33ab701f36c8fec9fe99c87c52307fc3eddb7"} err="failed to get container status \"4b71cf7c808ac46bbd87a5430dc33ab701f36c8fec9fe99c87c52307fc3eddb7\": rpc error: code = NotFound desc = could not find container \"4b71cf7c808ac46bbd87a5430dc33ab701f36c8fec9fe99c87c52307fc3eddb7\": container with ID starting with 4b71cf7c808ac46bbd87a5430dc33ab701f36c8fec9fe99c87c52307fc3eddb7 not found: ID does not exist" Nov 03 20:24:10.649304 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.649092 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-token" pod "6454d0c8-5104-4554-ab8d-4f822b372bb2" (UID: "6454d0c8-5104-4554-ab8d-4f822b372bb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.652003 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.651972 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-envoy" pod "6454d0c8-5104-4554-ab8d-4f822b372bb2" (UID: "6454d0c8-5104-4554-ab8d-4f822b372bb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:10.654643 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.654582 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b"] Nov 03 20:24:10.655149 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.655117 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-kube-api-access-fm5jw" pod "6454d0c8-5104-4554-ab8d-4f822b372bb2" (UID: "6454d0c8-5104-4554-ab8d-4f822b372bb2"). InnerVolumeSpecName "kube-api-access-fm5jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:10.664255 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.664230 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-5sk6b"] Nov 03 20:24:10.735852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.735688 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6454d0c8-5104-4554-ab8d-4f822b372bb2-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.735852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.735729 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm5jw\" (UniqueName: \"kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-kube-api-access-fm5jw\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.735852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.735744 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.735852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.735759 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.735852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.735775 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.735852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.735789 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.735852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.735802 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.735852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.735815 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6454d0c8-5104-4554-ab8d-4f822b372bb2-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:10.735852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:10.735827 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6454d0c8-5104-4554-ab8d-4f822b372bb2-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:11.572413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:11.572103 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" Nov 03 20:24:11.572413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:11.572139 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk" event={"ID":"6454d0c8-5104-4554-ab8d-4f822b372bb2","Type":"ContainerDied","Data":"0757e75011973159503a4c38aa50cc38b42677688141cdda6979d7482d7e6094"} Nov 03 20:24:11.572413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:11.572197 4457 scope.go:122] "RemoveContainer" containerID="e2f0e5ab4cf200747edd792160556fc2e662b7d0e6edf81652ac8bcf900882fe" Nov 03 20:24:11.689877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:11.687461 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk"] Nov 03 20:24:11.740164 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:11.718875 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-15e02e6b-openshift-default-5559dd5449-wfjdk"] Nov 03 20:24:11.870465 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:11.869618 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" path="/var/lib/kubelet/pods/1f5ce6fb-4dca-4cd7-befa-82ea92a023b1/volumes" Nov 03 20:24:11.874403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:11.870674 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6454d0c8-5104-4554-ab8d-4f822b372bb2" path="/var/lib/kubelet/pods/6454d0c8-5104-4554-ab8d-4f822b372bb2/volumes" Nov 03 20:24:11.874403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:11.871578 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" path="/var/lib/kubelet/pods/cc55dea0-c60e-43f3-b4d1-43b0f9486226/volumes" Nov 03 20:24:15.503103 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.503057 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t"] Nov 03 20:24:15.504296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.504271 4457 cpu_manager.go:414] "containerMap: removing container" podUID="1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" containerName="istio-proxy" Nov 03 20:24:15.504296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.504297 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" containerName="istio-proxy" Nov 03 20:24:15.504522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.504319 4457 cpu_manager.go:414] "containerMap: removing container" podUID="6454d0c8-5104-4554-ab8d-4f822b372bb2" containerName="istio-proxy" Nov 03 20:24:15.504522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.504326 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6454d0c8-5104-4554-ab8d-4f822b372bb2" containerName="istio-proxy" Nov 03 20:24:15.504522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.504359 4457 cpu_manager.go:414] "containerMap: removing container" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerName="istio-proxy" Nov 03 20:24:15.504522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.504367 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerName="istio-proxy" Nov 03 20:24:15.504694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.504609 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f5ce6fb-4dca-4cd7-befa-82ea92a023b1" containerName="istio-proxy" Nov 03 20:24:15.504694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.504624 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="6454d0c8-5104-4554-ab8d-4f822b372bb2" containerName="istio-proxy" Nov 03 20:24:15.504694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.504639 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc55dea0-c60e-43f3-b4d1-43b0f9486226" containerName="istio-proxy" Nov 03 20:24:15.511531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.511501 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.519577 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.519552 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t"] Nov 03 20:24:15.596034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.596001 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.596232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.596040 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.596232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.596156 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.596356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.596271 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.596356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.596314 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.596476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.596367 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.596538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.596478 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d44a0d-26ce-4622-b30f-f2c6e013828b-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.596538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.596522 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9v65\" (UniqueName: \"kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-kube-api-access-n9v65\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.596647 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.596569 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.597800 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.597780 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5ca474-openshift-default-dockercfg-p4nhm\"" Nov 03 20:24:15.698286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.698245 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.698575 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.698546 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.698711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.698645 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.698779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.698743 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.698844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.698822 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.698904 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.698865 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.698975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.698948 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.699058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.699041 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d44a0d-26ce-4622-b30f-f2c6e013828b-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.699112 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.699093 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9v65\" (UniqueName: \"kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-kube-api-access-n9v65\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.699174 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.699159 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.699556 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.699532 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.699653 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.699041 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.699653 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.699587 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.700184 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.700161 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d44a0d-26ce-4622-b30f-f2c6e013828b-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.708442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.702231 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.708442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.704914 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.718565 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.718542 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.718774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.718752 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9v65\" (UniqueName: \"kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-kube-api-access-n9v65\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:15.849997 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:15.849963 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:16.168133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:16.167517 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t"] Nov 03 20:24:16.170440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:16.170362 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:16.170518 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:16.170483 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:16.170576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:16.170528 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:16.640288 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:16.640235 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" event={"ID":"38d44a0d-26ce-4622-b30f-f2c6e013828b","Type":"ContainerStarted","Data":"2f09ffd899f4c501dc266e1e206b2ad12db4c4fd2f12a4ce84aea172aaa09966"} Nov 03 20:24:16.640878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:16.640850 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" event={"ID":"38d44a0d-26ce-4622-b30f-f2c6e013828b","Type":"ContainerStarted","Data":"5a92b0312e0807f1e62a735883b7819090697a0a25f33c58deb2453969ffdf9b"} Nov 03 20:24:16.672218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:16.672150 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" podStartSLOduration=1.672131096 podStartE2EDuration="1.672131096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:16.663986457 +0000 UTC m=+1635.039583207" watchObservedRunningTime="2026-11-03 20:24:16.672131096 +0000 UTC m=+1635.047727844" Nov 03 20:24:16.850366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:16.850336 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:17.857663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:17.857613 4457 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 20:24:17.858312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:17.858261 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" podUID="38d44a0d-26ce-4622-b30f-f2c6e013828b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 20:24:18.522414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:18.521473 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" Nov 03 20:24:18.522837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:18.522810 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" Nov 03 20:24:18.851116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:18.851084 4457 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 20:24:18.851267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:18.851157 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" podUID="38d44a0d-26ce-4622-b30f-f2c6e013828b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 20:24:19.850428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:19.850380 4457 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 20:24:19.850885 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:19.850455 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" podUID="38d44a0d-26ce-4622-b30f-f2c6e013828b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 20:24:20.542616 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.542582 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6"] Nov 03 20:24:20.573411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.559882 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.588455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.584567 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6cf8ae-openshift-default-dockercfg-db88k\"" Nov 03 20:24:20.627406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.626604 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6"] Nov 03 20:24:20.684409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.656190 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-data\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.692411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.684778 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4q52\" (UniqueName: \"kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-kube-api-access-t4q52\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.692698 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.692667 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.692948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.692923 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-certs\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.693086 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.693068 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-token\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.693265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.693250 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-podinfo\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.695099 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.695072 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istiod-ca-cert\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.695309 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.695292 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-credential-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.695485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.695467 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-envoy\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.797723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.797687 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.797971 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.797947 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-certs\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.798075 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.798021 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-token\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.798144 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.798103 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-podinfo\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.798203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.798162 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.798573 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.798541 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-certs\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.799479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.798760 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istiod-ca-cert\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.799614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.799530 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-credential-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.799614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.799590 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-envoy\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.799729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.799676 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-data\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.799799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.799753 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4q52\" (UniqueName: \"kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-kube-api-access-t4q52\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.800037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.799420 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istiod-ca-cert\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.800178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.800160 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-data\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.800361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.800332 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-credential-socket\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.801682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.801235 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-podinfo\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.802058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.802037 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-envoy\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.810615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.810591 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-token\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.814078 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.814057 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4q52\" (UniqueName: \"kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-kube-api-access-t4q52\") pod \"e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:20.850481 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.850448 4457 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 20:24:20.850794 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.850495 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" podUID="38d44a0d-26ce-4622-b30f-f2c6e013828b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 20:24:20.939725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:20.939687 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:21.295578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:21.292710 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6"] Nov 03 20:24:21.307161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:21.307131 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:21.307329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:21.307308 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:21.307459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:21.307445 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:21.820964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:21.820929 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" event={"ID":"e8bcbefd-88d8-4eae-9cf5-fa29620724a2","Type":"ContainerStarted","Data":"65b023e3e9738b2de64fd05f238c8b0e5bdb2efca4f00efd644f6af140f51850"} Nov 03 20:24:21.821147 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:21.820990 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" event={"ID":"e8bcbefd-88d8-4eae-9cf5-fa29620724a2","Type":"ContainerStarted","Data":"17ce3395795541d50bd6a283acc6baf77c80fe20b1967060e575b3d27682c594"} Nov 03 20:24:21.850773 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:21.850734 4457 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 20:24:21.851222 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:21.850783 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" podUID="38d44a0d-26ce-4622-b30f-f2c6e013828b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 20:24:21.851347 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:21.851302 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" podStartSLOduration=1.851284921 podStartE2EDuration="1.851284921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:21.844979547 +0000 UTC m=+1640.220576294" watchObservedRunningTime="2026-11-03 20:24:21.851284921 +0000 UTC m=+1640.226881668" Nov 03 20:24:21.939978 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:21.939948 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:22.850302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:22.850266 4457 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Nov 03 20:24:22.850530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:22.850338 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" podUID="38d44a0d-26ce-4622-b30f-f2c6e013828b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Nov 03 20:24:22.941451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:22.941415 4457 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:24:22.941861 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:22.941468 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" podUID="e8bcbefd-88d8-4eae-9cf5-fa29620724a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 03 20:24:23.864629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:23.864596 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:23.876413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:23.867777 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:23.892623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:23.891655 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:24:23.941028 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:23.940997 4457 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:24:23.941185 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:23.941056 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" podUID="e8bcbefd-88d8-4eae-9cf5-fa29620724a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 03 20:24:24.941546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:24.941513 4457 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:24:24.942027 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:24.941580 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" podUID="e8bcbefd-88d8-4eae-9cf5-fa29620724a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 03 20:24:25.942775 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:25.942745 4457 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:24:25.943256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:25.943230 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" podUID="e8bcbefd-88d8-4eae-9cf5-fa29620724a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 03 20:24:26.940562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:26.940529 4457 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:24:26.940751 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:26.940586 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" podUID="e8bcbefd-88d8-4eae-9cf5-fa29620724a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 03 20:24:27.940454 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:27.940375 4457 patch_prober.go:29] interesting pod/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Nov 03 20:24:27.940845 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:27.940471 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" podUID="e8bcbefd-88d8-4eae-9cf5-fa29620724a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Nov 03 20:24:28.945539 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:28.945502 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:28.947205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:28.947184 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:28.947551 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:28.947527 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:24:31.007396 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.007340 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55"] Nov 03 20:24:31.020261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.020229 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55"] Nov 03 20:24:31.020458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.020330 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" Nov 03 20:24:31.113011 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.112919 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:24:31.113224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.112981 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55\" (UID: \"970d32a1-7723-49cf-9378-8013c7ae9d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" Nov 03 20:24:31.113746 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.113403 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:24:31.113746 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.113604 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55\" (UID: \"970d32a1-7723-49cf-9378-8013c7ae9d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" Nov 03 20:24:31.113907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.113869 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv7n4\" (UniqueName: \"kubernetes.io/projected/970d32a1-7723-49cf-9378-8013c7ae9d46-kube-api-access-qv7n4\") pod \"maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55\" (UID: \"970d32a1-7723-49cf-9378-8013c7ae9d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" Nov 03 20:24:31.387042 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.387007 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx"] Nov 03 20:24:31.485453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.463044 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55\" (UID: \"970d32a1-7723-49cf-9378-8013c7ae9d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" Nov 03 20:24:31.485453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.463597 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv7n4\" (UniqueName: \"kubernetes.io/projected/970d32a1-7723-49cf-9378-8013c7ae9d46-kube-api-access-qv7n4\") pod \"maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55\" (UID: \"970d32a1-7723-49cf-9378-8013c7ae9d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" Nov 03 20:24:31.485453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.463660 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55\" (UID: \"970d32a1-7723-49cf-9378-8013c7ae9d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" Nov 03 20:24:31.504289 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.504246 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55\" (UID: \"970d32a1-7723-49cf-9378-8013c7ae9d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" Nov 03 20:24:31.665526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.665420 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv7n4\" (UniqueName: \"kubernetes.io/projected/970d32a1-7723-49cf-9378-8013c7ae9d46-kube-api-access-qv7n4\") pod \"maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55\" (UID: \"970d32a1-7723-49cf-9378-8013c7ae9d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" Nov 03 20:24:31.668998 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.668860 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55\" (UID: \"970d32a1-7723-49cf-9378-8013c7ae9d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" Nov 03 20:24:31.816408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.813049 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" Nov 03 20:24:31.829410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.828552 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:31.935033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.933870 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-34e844\"/\"kube-root-ca.crt\"" Nov 03 20:24:31.935033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.934589 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-34e844\"/\"default-dockercfg-24cw4\"" Nov 03 20:24:31.935033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.934681 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-34e844\"/\"openshift-service-ca.crt\"" Nov 03 20:24:31.935033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:31.934799 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-34e844\"/\"test-model-34e844-kserve-self-signed-certs\"" Nov 03 20:24:32.059356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.053526 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx"] Nov 03 20:24:32.116852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.116049 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-home\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.116852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.116125 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-model-cache\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.116852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.116334 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8988\" (UniqueName: \"kubernetes.io/projected/5d2da67e-e41e-441b-9a9d-1272bc5726a4-kube-api-access-m8988\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.116852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.116427 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-dshm\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.116852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.116488 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tls-certs\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.116852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.116600 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tmp-dir\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.223657 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.223564 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8988\" (UniqueName: \"kubernetes.io/projected/5d2da67e-e41e-441b-9a9d-1272bc5726a4-kube-api-access-m8988\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.223943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.223919 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-dshm\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.232686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.232655 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tls-certs\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.233525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.233502 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tmp-dir\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.234135 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.234119 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-home\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.238291 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.237078 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-home\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.238513 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.233980 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tmp-dir\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.257841 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.238671 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-model-cache\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.258229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.257746 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-model-cache\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.273615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.270497 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8988\" (UniqueName: \"kubernetes.io/projected/5d2da67e-e41e-441b-9a9d-1272bc5726a4-kube-api-access-m8988\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.378915 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.326412 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tls-certs\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.494824 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.474091 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-dshm\") pod \"test-model-34e844-kserve-6945d4458c-4rjtx\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.494824 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.478658 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:32.749208 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:32.749085 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55"] Nov 03 20:24:33.100697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:33.100661 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx"] Nov 03 20:24:33.152096 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:33.152050 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" event={"ID":"970d32a1-7723-49cf-9378-8013c7ae9d46","Type":"ContainerStarted","Data":"1d87a7bf4a24d5e79d7c22ef1422688efe743d991700e2d52ec2248a81133975"} Nov 03 20:24:34.253412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:34.253052 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" event={"ID":"5d2da67e-e41e-441b-9a9d-1272bc5726a4","Type":"ContainerStarted","Data":"7437236e4643c954e24c4946b461905d569c6cee339db97f79a9653e89207ec1"} Nov 03 20:24:34.253412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:34.253127 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" event={"ID":"5d2da67e-e41e-441b-9a9d-1272bc5726a4","Type":"ContainerStarted","Data":"c7a60dbae27829718d43a3703fb54228a544c1c7b559e2662b669f3a58990e9f"} Nov 03 20:24:34.268407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:34.265294 4457 generic.go:356] "Generic (PLEG): container finished" podID="970d32a1-7723-49cf-9378-8013c7ae9d46" containerID="7d40943aad01781caad8c08fa2423b2cc04bfebf682487b86da0899c410e2a40" exitCode=0 Nov 03 20:24:34.268407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:34.265527 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" event={"ID":"970d32a1-7723-49cf-9378-8013c7ae9d46","Type":"ContainerDied","Data":"7d40943aad01781caad8c08fa2423b2cc04bfebf682487b86da0899c410e2a40"} Nov 03 20:24:34.288411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:34.287021 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" podStartSLOduration=3.287004258 podStartE2EDuration="3.287004258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:34.273217905 +0000 UTC m=+1652.648814652" watchObservedRunningTime="2026-11-03 20:24:34.287004258 +0000 UTC m=+1652.662601008" Nov 03 20:24:34.490469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:34.490128 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:35.066410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:35.037689 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:35.066410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:35.045376 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:35.174289 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:35.173026 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:35.188409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:35.187488 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:35.212411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:35.209655 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:35.235412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:35.234583 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:35.242446 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:35.240691 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:35.258420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:35.257503 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:35.875704 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:35.875604 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:24:35.902013 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:35.900011 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/health\": context deadline exceeded" Nov 03 20:24:35.920429 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:35.909511 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.025960 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.025484 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded" Nov 03 20:24:36.045585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.045520 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.066196 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.066142 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.155883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.136930 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:24:36.258409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.253623 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:24:36.258409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.255208 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:24:36.260514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.260482 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.262069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.262049 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:24:36.286462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.278848 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" Nov 03 20:24:36.302930 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.302197 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.302930 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.302701 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:24:36.391433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.390560 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:24:36.454624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.452976 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:24:36.487839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.487812 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" Nov 03 20:24:36.488176 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.488151 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55" event={"ID":"970d32a1-7723-49cf-9378-8013c7ae9d46","Type":"ContainerDied","Data":"1d87a7bf4a24d5e79d7c22ef1422688efe743d991700e2d52ec2248a81133975"} Nov 03 20:24:36.488320 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.488303 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d87a7bf4a24d5e79d7c22ef1422688efe743d991700e2d52ec2248a81133975" Nov 03 20:24:36.526554 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.526138 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded" Nov 03 20:24:36.539482 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:24:36.539219 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:24:36.581621 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.581575 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.666533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.643307 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.698031 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.697973 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-openshift-ca\") pod \"970d32a1-7723-49cf-9378-8013c7ae9d46\" (UID: \"970d32a1-7723-49cf-9378-8013c7ae9d46\") " Nov 03 20:24:36.700696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.700667 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/970d32a1-7723-49cf-9378-8013c7ae9d46-kube-api-access-qv7n4\" (UniqueName: \"kubernetes.io/projected/970d32a1-7723-49cf-9378-8013c7ae9d46-kube-api-access-qv7n4\") pod \"970d32a1-7723-49cf-9378-8013c7ae9d46\" (UID: \"970d32a1-7723-49cf-9378-8013c7ae9d46\") " Nov 03 20:24:36.700970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.700951 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-xks-ca\") pod \"970d32a1-7723-49cf-9378-8013c7ae9d46\" (UID: \"970d32a1-7723-49cf-9378-8013c7ae9d46\") " Nov 03 20:24:36.704616 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.703858 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-openshift-ca" pod "970d32a1-7723-49cf-9378-8013c7ae9d46" (UID: "970d32a1-7723-49cf-9378-8013c7ae9d46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:36.874413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.824340 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:36.874413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.869943 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.896229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.879462 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:36.896229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.886202 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:24:37.000661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.966627 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-xks-ca" pod "970d32a1-7723-49cf-9378-8013c7ae9d46" (UID: "970d32a1-7723-49cf-9378-8013c7ae9d46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:24:37.029511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:36.992976 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:24:37.042012 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:37.041983 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:24:37.090655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:37.090612 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": context deadline exceeded" Nov 03 20:24:37.120643 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:37.092635 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970d32a1-7723-49cf-9378-8013c7ae9d46-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:37.120880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:37.120503 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:24:37.320409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:37.302878 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970d32a1-7723-49cf-9378-8013c7ae9d46-kube-api-access-qv7n4" pod "970d32a1-7723-49cf-9378-8013c7ae9d46" (UID: "970d32a1-7723-49cf-9378-8013c7ae9d46"). InnerVolumeSpecName "kube-api-access-qv7n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:37.320409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:37.303713 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded" Nov 03 20:24:37.330305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:37.330255 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv7n4\" (UniqueName: \"kubernetes.io/projected/970d32a1-7723-49cf-9378-8013c7ae9d46-kube-api-access-qv7n4\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:37.585474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:37.574950 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:24:37.978196 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:37.978162 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:24:41.155451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:41.152509 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x"] Nov 03 20:24:41.167895 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:41.167838 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc"] Nov 03 20:24:42.481634 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:42.480527 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:42.496858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:42.496829 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:42.984405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:42.984353 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:24:43.244868 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.244776 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq"] Nov 03 20:24:43.246062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.246031 4457 cpu_manager.go:414] "containerMap: removing container" podUID="970d32a1-7723-49cf-9378-8013c7ae9d46" containerName="revoke-keys" Nov 03 20:24:43.246062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.246060 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="970d32a1-7723-49cf-9378-8013c7ae9d46" containerName="revoke-keys" Nov 03 20:24:43.246335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.246317 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="970d32a1-7723-49cf-9378-8013c7ae9d46" containerName="revoke-keys" Nov 03 20:24:43.252494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.252464 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:43.270109 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.270076 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8c684d-serving-cert\"" Nov 03 20:24:43.285489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.285460 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq"] Nov 03 20:24:43.387190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.387088 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcwbz\" (UniqueName: \"kubernetes.io/projected/f67ad7d6-ee78-41e7-9644-81e1989198c1-kube-api-access-zcwbz\") pod \"maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:43.387520 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.387496 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-metrics-certs\") pod \"maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:43.388058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.388037 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:43.388462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.388443 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67ad7d6-ee78-41e7-9644-81e1989198c1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:43.490531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.490485 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-metrics-certs\") pod \"maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:43.491221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.491175 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:43.491909 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:24:43.491871 4457 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-serving-cert: secret "maas-api-e2e-inf-a-8c684d-serving-cert" not found Nov 03 20:24:43.492071 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:24:43.491969 4457 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-maas-api-tls podName:f67ad7d6-ee78-41e7-9644-81e1989198c1 nodeName:}" failed. No retries permitted until 2026-11-03 20:24:43.991946155 +0000 UTC m=+1662.367542894 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-maas-api-tls") pod "maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" (UID: "f67ad7d6-ee78-41e7-9644-81e1989198c1") : secret "maas-api-e2e-inf-a-8c684d-serving-cert" not found Nov 03 20:24:43.492346 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.492268 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67ad7d6-ee78-41e7-9644-81e1989198c1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:43.492665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.492645 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcwbz\" (UniqueName: \"kubernetes.io/projected/f67ad7d6-ee78-41e7-9644-81e1989198c1-kube-api-access-zcwbz\") pod \"maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:43.493178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.493057 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67ad7d6-ee78-41e7-9644-81e1989198c1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:43.497417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.497340 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-metrics-certs\") pod \"maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:43.508434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:43.506410 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcwbz\" (UniqueName: \"kubernetes.io/projected/f67ad7d6-ee78-41e7-9644-81e1989198c1-kube-api-access-zcwbz\") pod \"maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:44.046412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:44.039792 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:44.088420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:44.087048 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-maas-api-tls\") pod \"maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:44.204312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:44.202683 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:44.709413 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:24:44.704702 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67ad7d6_ee78_41e7_9644_81e1989198c1.slice/crio-5b0ff6493437aac5b1f118fdcc66b7a87a47d6979eb473dffd30a0c023d2e975 WatchSource:0}: Error finding container 5b0ff6493437aac5b1f118fdcc66b7a87a47d6979eb473dffd30a0c023d2e975: Status 404 returned error can't find the container with id 5b0ff6493437aac5b1f118fdcc66b7a87a47d6979eb473dffd30a0c023d2e975 Nov 03 20:24:44.734408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:44.710060 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq"] Nov 03 20:24:45.082749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:45.082704 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" event={"ID":"f67ad7d6-ee78-41e7-9644-81e1989198c1","Type":"ContainerStarted","Data":"5b0ff6493437aac5b1f118fdcc66b7a87a47d6979eb473dffd30a0c023d2e975"} Nov 03 20:24:45.626469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:45.625952 4457 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:24:45.626469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:45.626026 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 20:24:46.098210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.098159 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" event={"ID":"f67ad7d6-ee78-41e7-9644-81e1989198c1","Type":"ContainerStarted","Data":"ee0406e57cda331a45f07c4961756e81a25ffcfa7fd3eb3edd8b8dc67e6e1d18"} Nov 03 20:24:46.098726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.098254 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:46.100628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.100597 4457 generic.go:356] "Generic (PLEG): container finished" podID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerID="372a03fdf78fdf92ee8b8db1f9918d1ac307245577fe1431ea379afc88676a45" exitCode=0 Nov 03 20:24:46.100765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.100667 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" event={"ID":"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2","Type":"ContainerDied","Data":"372a03fdf78fdf92ee8b8db1f9918d1ac307245577fe1431ea379afc88676a45"} Nov 03 20:24:46.101162 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.101143 4457 scope.go:122] "RemoveContainer" containerID="372a03fdf78fdf92ee8b8db1f9918d1ac307245577fe1431ea379afc88676a45" Nov 03 20:24:46.103662 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.103632 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:46.103747 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.103719 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:46.103787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.103765 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:24:46.125534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.125469 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" podStartSLOduration=2.668928564 podStartE2EDuration="3.125449945s" totalImagesPullingTime="456.521381ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:24:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:24:46.121697983 +0000 UTC m=+1664.497294730" watchObservedRunningTime="2026-11-03 20:24:46.125449945 +0000 UTC m=+1664.501046693" Nov 03 20:24:46.153303 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.153272 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" podUID="f16ca003-2cef-4af2-ba76-9a3f76629867" containerName="payload-processing" containerID="cri-o://446bb0972bbdb2454dc22e883b03e022dc7a78b2a1ce99b5f4a54b364cf81a6b" gracePeriod=25 Nov 03 20:24:46.169209 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.169164 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" podUID="69e6d47e-f7f8-4eda-8f02-8ddcb150300c" containerName="payload-pre-processing" containerID="cri-o://198c1240afc97822fc2b6d96f265ebe0d3562bee45119e94d90c582df46a3945" gracePeriod=25 Nov 03 20:24:46.636462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.635060 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" Nov 03 20:24:46.708874 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.708839 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" Nov 03 20:24:46.732295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.729596 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-kube-api-access-jhx9h\" (UniqueName: \"kubernetes.io/projected/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-kube-api-access-jhx9h\") pod \"69e6d47e-f7f8-4eda-8f02-8ddcb150300c\" (UID: \"69e6d47e-f7f8-4eda-8f02-8ddcb150300c\") " Nov 03 20:24:46.732295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.730245 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-plugins-config-volume\") pod \"69e6d47e-f7f8-4eda-8f02-8ddcb150300c\" (UID: \"69e6d47e-f7f8-4eda-8f02-8ddcb150300c\") " Nov 03 20:24:46.732295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.730779 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-plugins-config-volume" pod "69e6d47e-f7f8-4eda-8f02-8ddcb150300c" (UID: "69e6d47e-f7f8-4eda-8f02-8ddcb150300c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:46.732295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.730868 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16ca003-2cef-4af2-ba76-9a3f76629867-kube-api-access-ns22b\" (UniqueName: \"kubernetes.io/projected/f16ca003-2cef-4af2-ba76-9a3f76629867-kube-api-access-ns22b\") pod \"f16ca003-2cef-4af2-ba76-9a3f76629867\" (UID: \"f16ca003-2cef-4af2-ba76-9a3f76629867\") " Nov 03 20:24:46.732295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.730943 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16ca003-2cef-4af2-ba76-9a3f76629867-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16ca003-2cef-4af2-ba76-9a3f76629867-plugins-config-volume\") pod \"f16ca003-2cef-4af2-ba76-9a3f76629867\" (UID: \"f16ca003-2cef-4af2-ba76-9a3f76629867\") " Nov 03 20:24:46.732295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.731602 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16ca003-2cef-4af2-ba76-9a3f76629867-plugins-config-volume" pod "f16ca003-2cef-4af2-ba76-9a3f76629867" (UID: "f16ca003-2cef-4af2-ba76-9a3f76629867"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:46.733566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.733524 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:46.733566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.733549 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f16ca003-2cef-4af2-ba76-9a3f76629867-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:46.761691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.757557 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16ca003-2cef-4af2-ba76-9a3f76629867-kube-api-access-ns22b" pod "f16ca003-2cef-4af2-ba76-9a3f76629867" (UID: "f16ca003-2cef-4af2-ba76-9a3f76629867"). InnerVolumeSpecName "kube-api-access-ns22b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:46.816874 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.816833 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-kube-api-access-jhx9h" pod "69e6d47e-f7f8-4eda-8f02-8ddcb150300c" (UID: "69e6d47e-f7f8-4eda-8f02-8ddcb150300c"). InnerVolumeSpecName "kube-api-access-jhx9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:46.835409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.835356 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhx9h\" (UniqueName: \"kubernetes.io/projected/69e6d47e-f7f8-4eda-8f02-8ddcb150300c-kube-api-access-jhx9h\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:46.835650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:46.835618 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns22b\" (UniqueName: \"kubernetes.io/projected/f16ca003-2cef-4af2-ba76-9a3f76629867-kube-api-access-ns22b\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:47.116019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.115989 4457 generic.go:356] "Generic (PLEG): container finished" podID="69e6d47e-f7f8-4eda-8f02-8ddcb150300c" containerID="198c1240afc97822fc2b6d96f265ebe0d3562bee45119e94d90c582df46a3945" exitCode=0 Nov 03 20:24:47.116468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.116068 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" Nov 03 20:24:47.116468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.116068 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" event={"ID":"69e6d47e-f7f8-4eda-8f02-8ddcb150300c","Type":"ContainerDied","Data":"198c1240afc97822fc2b6d96f265ebe0d3562bee45119e94d90c582df46a3945"} Nov 03 20:24:47.116468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.116185 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc" event={"ID":"69e6d47e-f7f8-4eda-8f02-8ddcb150300c","Type":"ContainerDied","Data":"601e52b01032263fe159ac71db4b8a0da29e95560847bf7629646481fbe7fc1d"} Nov 03 20:24:47.116468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.116208 4457 scope.go:122] "RemoveContainer" containerID="198c1240afc97822fc2b6d96f265ebe0d3562bee45119e94d90c582df46a3945" Nov 03 20:24:47.118702 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.118674 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" event={"ID":"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2","Type":"ContainerStarted","Data":"f8319497a62ab056cb6747a32c5b3b8e957d7b3d7960535c5d5511f7c438bd5a"} Nov 03 20:24:47.120379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.120358 4457 generic.go:356] "Generic (PLEG): container finished" podID="f16ca003-2cef-4af2-ba76-9a3f76629867" containerID="446bb0972bbdb2454dc22e883b03e022dc7a78b2a1ce99b5f4a54b364cf81a6b" exitCode=0 Nov 03 20:24:47.120577 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.120424 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" event={"ID":"f16ca003-2cef-4af2-ba76-9a3f76629867","Type":"ContainerDied","Data":"446bb0972bbdb2454dc22e883b03e022dc7a78b2a1ce99b5f4a54b364cf81a6b"} Nov 03 20:24:47.120577 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.120470 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" Nov 03 20:24:47.120577 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.120473 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x" event={"ID":"f16ca003-2cef-4af2-ba76-9a3f76629867","Type":"ContainerDied","Data":"2e31c8f82fb0d464bf01c7dd555097900fc1edacd6a3eae392b7cb9c6040a060"} Nov 03 20:24:47.129296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.129279 4457 scope.go:122] "RemoveContainer" containerID="198c1240afc97822fc2b6d96f265ebe0d3562bee45119e94d90c582df46a3945" Nov 03 20:24:47.129677 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:24:47.129651 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198c1240afc97822fc2b6d96f265ebe0d3562bee45119e94d90c582df46a3945\": container with ID starting with 198c1240afc97822fc2b6d96f265ebe0d3562bee45119e94d90c582df46a3945 not found: ID does not exist" containerID="198c1240afc97822fc2b6d96f265ebe0d3562bee45119e94d90c582df46a3945" Nov 03 20:24:47.129780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.129691 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198c1240afc97822fc2b6d96f265ebe0d3562bee45119e94d90c582df46a3945"} err="failed to get container status \"198c1240afc97822fc2b6d96f265ebe0d3562bee45119e94d90c582df46a3945\": rpc error: code = NotFound desc = could not find container \"198c1240afc97822fc2b6d96f265ebe0d3562bee45119e94d90c582df46a3945\": container with ID starting with 198c1240afc97822fc2b6d96f265ebe0d3562bee45119e94d90c582df46a3945 not found: ID does not exist" Nov 03 20:24:47.129780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.129714 4457 scope.go:122] "RemoveContainer" containerID="446bb0972bbdb2454dc22e883b03e022dc7a78b2a1ce99b5f4a54b364cf81a6b" Nov 03 20:24:47.156166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.156140 4457 scope.go:122] "RemoveContainer" containerID="446bb0972bbdb2454dc22e883b03e022dc7a78b2a1ce99b5f4a54b364cf81a6b" Nov 03 20:24:47.156547 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:24:47.156520 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"446bb0972bbdb2454dc22e883b03e022dc7a78b2a1ce99b5f4a54b364cf81a6b\": container with ID starting with 446bb0972bbdb2454dc22e883b03e022dc7a78b2a1ce99b5f4a54b364cf81a6b not found: ID does not exist" containerID="446bb0972bbdb2454dc22e883b03e022dc7a78b2a1ce99b5f4a54b364cf81a6b" Nov 03 20:24:47.156643 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.156557 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"446bb0972bbdb2454dc22e883b03e022dc7a78b2a1ce99b5f4a54b364cf81a6b"} err="failed to get container status \"446bb0972bbdb2454dc22e883b03e022dc7a78b2a1ce99b5f4a54b364cf81a6b\": rpc error: code = NotFound desc = could not find container \"446bb0972bbdb2454dc22e883b03e022dc7a78b2a1ce99b5f4a54b364cf81a6b\": container with ID starting with 446bb0972bbdb2454dc22e883b03e022dc7a78b2a1ce99b5f4a54b364cf81a6b not found: ID does not exist" Nov 03 20:24:47.177765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.173693 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc"] Nov 03 20:24:47.181320 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.181286 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3ebde5-7b4d7475d7-qkttc"] Nov 03 20:24:47.188754 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.188702 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x"] Nov 03 20:24:47.195225 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.195202 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3ebde5-8bdc695d6-7hv7x"] Nov 03 20:24:47.433466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.431803 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:24:47.433466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.431934 4457 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:24:47.433466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.431982 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 20:24:47.794931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.794860 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69e6d47e-f7f8-4eda-8f02-8ddcb150300c" path="/var/lib/kubelet/pods/69e6d47e-f7f8-4eda-8f02-8ddcb150300c/volumes" Nov 03 20:24:47.795874 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:47.795842 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16ca003-2cef-4af2-ba76-9a3f76629867" path="/var/lib/kubelet/pods/f16ca003-2cef-4af2-ba76-9a3f76629867/volumes" Nov 03 20:24:48.308572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:48.308536 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn"] Nov 03 20:24:48.309366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:48.309306 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" podUID="2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" containerName="istio-proxy" containerID="cri-o://a66d83f00f3997b54a401d05dbd7d363d7900571f45991e670b1c91afb45b807" gracePeriod=30 Nov 03 20:24:48.430410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:48.429622 4457 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:24:48.430410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:48.429683 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 20:24:49.430299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:49.430260 4457 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:24:49.430691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:49.430326 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 20:24:50.433370 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:50.430766 4457 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:24:50.433370 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:50.430836 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 20:24:51.432960 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:51.432895 4457 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:24:51.433261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:51.433229 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 20:24:52.130267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:52.130211 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:24:52.429950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:52.429868 4457 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Nov 03 20:24:52.430166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:52.430141 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Nov 03 20:24:53.435224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.435192 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:24:53.435642 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.435583 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:24:53.436439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.436415 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:24:53.741010 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.740983 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:24:53.778409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.775353 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-kube-api-access-szk7m\" (UniqueName: \"kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-kube-api-access-szk7m\") pod \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " Nov 03 20:24:53.778409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.775505 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-envoy\") pod \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " Nov 03 20:24:53.778409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.775547 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-token\" (UniqueName: \"kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-token\") pod \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " Nov 03 20:24:53.778409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.775602 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-socket\") pod \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " Nov 03 20:24:53.778409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.775769 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-podinfo\") pod \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " Nov 03 20:24:53.781024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.779764 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-socket" pod "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" (UID: "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:53.789861 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.789806 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-kube-api-access-szk7m" pod "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" (UID: "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41"). InnerVolumeSpecName "kube-api-access-szk7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:53.792648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.792621 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-podinfo" pod "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" (UID: "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:24:53.796516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.796491 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-envoy" pod "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" (UID: "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:53.849925 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.847615 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-token" pod "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" (UID: "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:24:53.880420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.879810 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-certs\") pod \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " Nov 03 20:24:53.880420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.879862 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istiod-ca-cert\") pod \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " Nov 03 20:24:53.880420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.879979 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-credential-socket\") pod \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " Nov 03 20:24:53.880420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.880132 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-data\") pod \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\" (UID: \"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41\") " Nov 03 20:24:53.902461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.899348 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-data" pod "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" (UID: "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:53.902461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.899594 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-certs" pod "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" (UID: "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:53.902461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.899897 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istiod-ca-cert" pod "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" (UID: "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:24:53.902461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.900355 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-credential-socket" pod "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" (UID: "2f873bb5-5e1d-46fc-a4b6-6d13decdeb41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:24:53.902864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.902499 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.902864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.902536 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.902864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.902551 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.902864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.902566 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szk7m\" (UniqueName: \"kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-kube-api-access-szk7m\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.902864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.902582 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.902864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.902596 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.902864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.902610 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.902864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.902625 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:53.902864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:53.902640 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:24:54.248659 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:54.248621 4457 generic.go:356] "Generic (PLEG): container finished" podID="2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" containerID="a66d83f00f3997b54a401d05dbd7d363d7900571f45991e670b1c91afb45b807" exitCode=0 Nov 03 20:24:54.248852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:54.248684 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" Nov 03 20:24:54.248852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:54.248705 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" event={"ID":"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41","Type":"ContainerDied","Data":"a66d83f00f3997b54a401d05dbd7d363d7900571f45991e670b1c91afb45b807"} Nov 03 20:24:54.248852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:54.248763 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn" event={"ID":"2f873bb5-5e1d-46fc-a4b6-6d13decdeb41","Type":"ContainerDied","Data":"db58586a287ca574d120e537e8a7d9a50a13f19c26e254277359b1ca3af616ed"} Nov 03 20:24:54.248852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:54.248786 4457 scope.go:122] "RemoveContainer" containerID="a66d83f00f3997b54a401d05dbd7d363d7900571f45991e670b1c91afb45b807" Nov 03 20:24:54.269588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:54.269565 4457 scope.go:122] "RemoveContainer" containerID="a66d83f00f3997b54a401d05dbd7d363d7900571f45991e670b1c91afb45b807" Nov 03 20:24:54.269955 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:24:54.269907 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66d83f00f3997b54a401d05dbd7d363d7900571f45991e670b1c91afb45b807\": container with ID starting with a66d83f00f3997b54a401d05dbd7d363d7900571f45991e670b1c91afb45b807 not found: ID does not exist" containerID="a66d83f00f3997b54a401d05dbd7d363d7900571f45991e670b1c91afb45b807" Nov 03 20:24:54.270059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:54.269970 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66d83f00f3997b54a401d05dbd7d363d7900571f45991e670b1c91afb45b807"} err="failed to get container status \"a66d83f00f3997b54a401d05dbd7d363d7900571f45991e670b1c91afb45b807\": rpc error: code = NotFound desc = could not find container \"a66d83f00f3997b54a401d05dbd7d363d7900571f45991e670b1c91afb45b807\": container with ID starting with a66d83f00f3997b54a401d05dbd7d363d7900571f45991e670b1c91afb45b807 not found: ID does not exist" Nov 03 20:24:54.282533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:54.282500 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn"] Nov 03 20:24:54.289124 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:54.289099 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-x24zn"] Nov 03 20:24:55.761068 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:24:55.761036 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" path="/var/lib/kubelet/pods/2f873bb5-5e1d-46fc-a4b6-6d13decdeb41/volumes" Nov 03 20:25:00.563316 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.563276 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk"] Nov 03 20:25:00.564325 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.564301 4457 cpu_manager.go:414] "containerMap: removing container" podUID="f16ca003-2cef-4af2-ba76-9a3f76629867" containerName="payload-processing" Nov 03 20:25:00.564325 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.564326 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16ca003-2cef-4af2-ba76-9a3f76629867" containerName="payload-processing" Nov 03 20:25:00.564557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.564344 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" containerName="istio-proxy" Nov 03 20:25:00.564557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.564352 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" containerName="istio-proxy" Nov 03 20:25:00.564557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.564374 4457 cpu_manager.go:414] "containerMap: removing container" podUID="69e6d47e-f7f8-4eda-8f02-8ddcb150300c" containerName="payload-pre-processing" Nov 03 20:25:00.564557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.564382 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69e6d47e-f7f8-4eda-8f02-8ddcb150300c" containerName="payload-pre-processing" Nov 03 20:25:00.564756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.564685 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16ca003-2cef-4af2-ba76-9a3f76629867" containerName="payload-processing" Nov 03 20:25:00.564756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.564703 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f873bb5-5e1d-46fc-a4b6-6d13decdeb41" containerName="istio-proxy" Nov 03 20:25:00.564859 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.564717 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="69e6d47e-f7f8-4eda-8f02-8ddcb150300c" containerName="payload-pre-processing" Nov 03 20:25:00.569721 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.569698 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.592318 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.592289 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk"] Nov 03 20:25:00.706111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.706071 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.706282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.706151 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.706323 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.706276 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlcq6\" (UniqueName: \"kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-kube-api-access-qlcq6\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.706323 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.706315 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.706487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.706472 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.706608 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.706589 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.706648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.706632 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.706695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.706678 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.706789 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.706775 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.808646 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.808610 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.808827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.808702 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.808827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.808818 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlcq6\" (UniqueName: \"kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-kube-api-access-qlcq6\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.808951 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.808857 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.809012 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.808948 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.809080 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.809060 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.809080 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.809070 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-certs\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.809181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.809108 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.809181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.809142 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-data\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.809286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.809179 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.809286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.809250 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.809400 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.809332 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.809463 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.809449 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-credential-socket\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.809914 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.809889 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istiod-ca-cert\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.811585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.811561 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-envoy\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.811716 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.811698 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-podinfo\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.826417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.826334 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-token\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.826628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.826607 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlcq6\" (UniqueName: \"kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-kube-api-access-qlcq6\") pod \"e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:00.896024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:00.895983 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:01.195778 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:25:01.195749 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ca19a7_29db_4b9c_a0fd_5f75e1bf0e18.slice/crio-de38d172fd1bf797cdb1334aca0969facf9add070cee581a42355432697a7c10 WatchSource:0}: Error finding container de38d172fd1bf797cdb1334aca0969facf9add070cee581a42355432697a7c10: Status 404 returned error can't find the container with id de38d172fd1bf797cdb1334aca0969facf9add070cee581a42355432697a7c10 Nov 03 20:25:01.199082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:01.199044 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:01.199200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:01.199136 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:01.199200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:01.199183 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:01.200753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:01.200726 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk"] Nov 03 20:25:01.354639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:01.354601 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" event={"ID":"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18","Type":"ContainerStarted","Data":"abf13da326034ef676cc116886c83ad68f0b07e56860a51b9d1f504c018119db"} Nov 03 20:25:01.354855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:01.354661 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" event={"ID":"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18","Type":"ContainerStarted","Data":"de38d172fd1bf797cdb1334aca0969facf9add070cee581a42355432697a7c10"} Nov 03 20:25:01.383569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:01.383495 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" podStartSLOduration=1.383476339 podStartE2EDuration="1.383476339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:01.377481211 +0000 UTC m=+1679.753077960" watchObservedRunningTime="2026-11-03 20:25:01.383476339 +0000 UTC m=+1679.759073084" Nov 03 20:25:01.918549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:01.918475 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:02.900156 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:02.900120 4457 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:25:02.900340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:02.900170 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" podUID="06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 03 20:25:03.897409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:03.896801 4457 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:25:03.897409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:03.896866 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" podUID="06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 03 20:25:04.898023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:04.897524 4457 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:25:04.898023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:04.897596 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" podUID="06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 03 20:25:05.896935 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:05.896893 4457 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:25:05.897145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:05.896974 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" podUID="06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 03 20:25:06.148346 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:25:06.148248 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:25:06.711693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:06.711658 4457 scope.go:122] "RemoveContainer" containerID="89904689beceaf5a1e9929e6cccc9d91eeddf5c65db67dfc70958b0600d72440" Nov 03 20:25:06.723449 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:06.723425 4457 scope.go:122] "RemoveContainer" containerID="f48fef060688540d37cbbf95e84cb80d0e2eccb124aa109d6851b47726d103c6" Nov 03 20:25:06.918619 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:06.915732 4457 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:25:06.918619 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:06.915797 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" podUID="06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 03 20:25:06.924642 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:06.919956 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6"] Nov 03 20:25:06.932548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:06.932521 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6"] Nov 03 20:25:06.932697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:06.932594 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" Nov 03 20:25:06.934898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:06.934872 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk"] Nov 03 20:25:06.946871 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:06.939728 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" Nov 03 20:25:06.946871 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:06.941744 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d0898b\"" Nov 03 20:25:06.946871 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:06.941938 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d0898b-dockercfg-zcftk\"" Nov 03 20:25:06.950241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:06.950216 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk"] Nov 03 20:25:07.176485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.173617 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47128d77-8e6f-4e60-9299-585cfe98b268-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6\" (UID: \"47128d77-8e6f-4e60-9299-585cfe98b268\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" Nov 03 20:25:07.203569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.199525 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c24gb\" (UniqueName: \"kubernetes.io/projected/47128d77-8e6f-4e60-9299-585cfe98b268-kube-api-access-c24gb\") pod \"payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6\" (UID: \"47128d77-8e6f-4e60-9299-585cfe98b268\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" Nov 03 20:25:07.203569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.199628 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk\" (UID: \"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" Nov 03 20:25:07.203569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.199790 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkp5\" (UniqueName: \"kubernetes.io/projected/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-kube-api-access-zhkp5\") pod \"payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk\" (UID: \"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" Nov 03 20:25:07.427314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.421703 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47128d77-8e6f-4e60-9299-585cfe98b268-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6\" (UID: \"47128d77-8e6f-4e60-9299-585cfe98b268\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" Nov 03 20:25:07.427314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.422028 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c24gb\" (UniqueName: \"kubernetes.io/projected/47128d77-8e6f-4e60-9299-585cfe98b268-kube-api-access-c24gb\") pod \"payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6\" (UID: \"47128d77-8e6f-4e60-9299-585cfe98b268\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" Nov 03 20:25:07.427314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.422120 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk\" (UID: \"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" Nov 03 20:25:07.427314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.422261 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkp5\" (UniqueName: \"kubernetes.io/projected/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-kube-api-access-zhkp5\") pod \"payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk\" (UID: \"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" Nov 03 20:25:07.427314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.425680 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47128d77-8e6f-4e60-9299-585cfe98b268-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6\" (UID: \"47128d77-8e6f-4e60-9299-585cfe98b268\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" Nov 03 20:25:07.427314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.426941 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk\" (UID: \"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" Nov 03 20:25:07.439947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.438722 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c24gb\" (UniqueName: \"kubernetes.io/projected/47128d77-8e6f-4e60-9299-585cfe98b268-kube-api-access-c24gb\") pod \"payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6\" (UID: \"47128d77-8e6f-4e60-9299-585cfe98b268\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" Nov 03 20:25:07.440473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.440444 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkp5\" (UniqueName: \"kubernetes.io/projected/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-kube-api-access-zhkp5\") pod \"payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk\" (UID: \"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" Nov 03 20:25:07.513409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.505682 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" Nov 03 20:25:07.694803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.694714 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" Nov 03 20:25:07.930220 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.930109 4457 patch_prober.go:29] interesting pod/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" start-of-body= Nov 03 20:25:07.930544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:07.930512 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" podUID="06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.182:15021/healthz/ready\": dial tcp 10.133.0.182:15021: connect: connection refused" Nov 03 20:25:08.259656 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:25:08.259615 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47128d77_8e6f_4e60_9299_585cfe98b268.slice/crio-1621e400114baea2ec538e4dbceceadfc5d21ff466065dd3d1f51f57697abbfa WatchSource:0}: Error finding container 1621e400114baea2ec538e4dbceceadfc5d21ff466065dd3d1f51f57697abbfa: Status 404 returned error can't find the container with id 1621e400114baea2ec538e4dbceceadfc5d21ff466065dd3d1f51f57697abbfa Nov 03 20:25:08.268585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:08.268451 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6"] Nov 03 20:25:08.303491 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:25:08.303456 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2698e3c6_1dfd_4e2b_9d88_02dbee3f64e5.slice/crio-4997a3a871b65f08e498da5cde97c96c42e309053ca299f35452e68b82d69d5f WatchSource:0}: Error finding container 4997a3a871b65f08e498da5cde97c96c42e309053ca299f35452e68b82d69d5f: Status 404 returned error can't find the container with id 4997a3a871b65f08e498da5cde97c96c42e309053ca299f35452e68b82d69d5f Nov 03 20:25:08.306258 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:08.306231 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk"] Nov 03 20:25:08.688878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:08.688835 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" event={"ID":"47128d77-8e6f-4e60-9299-585cfe98b268","Type":"ContainerStarted","Data":"ab4a7020c483e3e0f5099e44cc3a94718cc1094aac4a7ac1e2a58c918523a224"} Nov 03 20:25:08.689056 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:08.688910 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" event={"ID":"47128d77-8e6f-4e60-9299-585cfe98b268","Type":"ContainerStarted","Data":"1621e400114baea2ec538e4dbceceadfc5d21ff466065dd3d1f51f57697abbfa"} Nov 03 20:25:08.689484 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:08.689464 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" Nov 03 20:25:08.692070 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:08.692038 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" event={"ID":"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5","Type":"ContainerStarted","Data":"60b1925b30e045fe0a5a8713c8f9335b7b491e1b1593fcad57414712cc39ac4e"} Nov 03 20:25:08.692196 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:08.692093 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" event={"ID":"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5","Type":"ContainerStarted","Data":"4997a3a871b65f08e498da5cde97c96c42e309053ca299f35452e68b82d69d5f"} Nov 03 20:25:08.692570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:08.692547 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" Nov 03 20:25:08.709263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:08.709213 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" podStartSLOduration=2.709200788 podStartE2EDuration="2.709200788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:08.708318012 +0000 UTC m=+1687.083914760" watchObservedRunningTime="2026-11-03 20:25:08.709200788 +0000 UTC m=+1687.084797535" Nov 03 20:25:08.728728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:08.728664 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" podStartSLOduration=2.72864744 podStartE2EDuration="2.72864744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:08.726599379 +0000 UTC m=+1687.102196128" watchObservedRunningTime="2026-11-03 20:25:08.72864744 +0000 UTC m=+1687.104244186" Nov 03 20:25:08.904080 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:08.904051 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:08.904827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:08.904802 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:08.906548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:08.906525 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:25:18.275827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:18.275789 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq"] Nov 03 20:25:18.306785 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:18.306745 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq"] Nov 03 20:25:18.306976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:18.306870 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" Nov 03 20:25:18.315720 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:18.315695 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b0adda-dockercfg-7tvrz\"" Nov 03 20:25:18.382423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:18.380725 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b0adda\"" Nov 03 20:25:18.439414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:18.438710 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e57f420-40a0-4f1b-8b58-70f0be591ec8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq\" (UID: \"7e57f420-40a0-4f1b-8b58-70f0be591ec8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" Nov 03 20:25:18.439414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:18.438971 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mj5t\" (UniqueName: \"kubernetes.io/projected/7e57f420-40a0-4f1b-8b58-70f0be591ec8-kube-api-access-2mj5t\") pod \"payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq\" (UID: \"7e57f420-40a0-4f1b-8b58-70f0be591ec8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" Nov 03 20:25:18.559284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:18.548585 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e57f420-40a0-4f1b-8b58-70f0be591ec8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq\" (UID: \"7e57f420-40a0-4f1b-8b58-70f0be591ec8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" Nov 03 20:25:18.559284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:18.548808 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mj5t\" (UniqueName: \"kubernetes.io/projected/7e57f420-40a0-4f1b-8b58-70f0be591ec8-kube-api-access-2mj5t\") pod \"payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq\" (UID: \"7e57f420-40a0-4f1b-8b58-70f0be591ec8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" Nov 03 20:25:18.559284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:18.550264 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e57f420-40a0-4f1b-8b58-70f0be591ec8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq\" (UID: \"7e57f420-40a0-4f1b-8b58-70f0be591ec8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" Nov 03 20:25:18.577717 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:18.576492 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mj5t\" (UniqueName: \"kubernetes.io/projected/7e57f420-40a0-4f1b-8b58-70f0be591ec8-kube-api-access-2mj5t\") pod \"payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq\" (UID: \"7e57f420-40a0-4f1b-8b58-70f0be591ec8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" Nov 03 20:25:18.637413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:18.631804 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" Nov 03 20:25:18.969315 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:18.968427 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq"] Nov 03 20:25:19.706138 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:19.706111 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" Nov 03 20:25:19.706675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:19.706177 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" Nov 03 20:25:19.887936 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:19.887886 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" event={"ID":"7e57f420-40a0-4f1b-8b58-70f0be591ec8","Type":"ContainerStarted","Data":"d20c1f13b2783e6999c0019d611fed0d580684674f82bf2c36372d162d9bcd01"} Nov 03 20:25:19.888160 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:19.887974 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" Nov 03 20:25:19.888160 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:19.887999 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" event={"ID":"7e57f420-40a0-4f1b-8b58-70f0be591ec8","Type":"ContainerStarted","Data":"54db25bd8e26be88d31f2bf23caf2958e74247487549d808526e2ed06376a7c3"} Nov 03 20:25:19.917571 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:19.917367 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" podStartSLOduration=1.917344149 podStartE2EDuration="1.917344149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:19.910594581 +0000 UTC m=+1698.286191329" watchObservedRunningTime="2026-11-03 20:25:19.917344149 +0000 UTC m=+1698.292940922" Nov 03 20:25:24.932745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:24.932711 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55"] Nov 03 20:25:24.940236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:24.940204 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3ebde5-8pl55"] Nov 03 20:25:25.766410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:25.765239 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="970d32a1-7723-49cf-9378-8013c7ae9d46" path="/var/lib/kubelet/pods/970d32a1-7723-49cf-9378-8013c7ae9d46/volumes" Nov 03 20:25:28.789601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.787862 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656"] Nov 03 20:25:28.806699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.806663 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" Nov 03 20:25:28.814899 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.814684 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:25:28.816397 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.816357 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:25:28.830222 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.830186 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656"] Nov 03 20:25:28.860226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.859753 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461f444c-pz656\" (UID: \"031f50c1-6a58-454b-a132-2291aa316cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" Nov 03 20:25:28.860226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.859853 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xsp2\" (UniqueName: \"kubernetes.io/projected/031f50c1-6a58-454b-a132-2291aa316cd3-kube-api-access-7xsp2\") pod \"maas-api-revoke-sub-461f444c-pz656\" (UID: \"031f50c1-6a58-454b-a132-2291aa316cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" Nov 03 20:25:28.863595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.862009 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461f444c-pz656\" (UID: \"031f50c1-6a58-454b-a132-2291aa316cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" Nov 03 20:25:28.964778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.964734 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461f444c-pz656\" (UID: \"031f50c1-6a58-454b-a132-2291aa316cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" Nov 03 20:25:28.965494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.965465 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461f444c-pz656\" (UID: \"031f50c1-6a58-454b-a132-2291aa316cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" Nov 03 20:25:28.966257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.966233 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461f444c-pz656\" (UID: \"031f50c1-6a58-454b-a132-2291aa316cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" Nov 03 20:25:28.966452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.966431 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xsp2\" (UniqueName: \"kubernetes.io/projected/031f50c1-6a58-454b-a132-2291aa316cd3-kube-api-access-7xsp2\") pod \"maas-api-revoke-sub-461f444c-pz656\" (UID: \"031f50c1-6a58-454b-a132-2291aa316cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" Nov 03 20:25:28.967612 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.967590 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461f444c-pz656\" (UID: \"031f50c1-6a58-454b-a132-2291aa316cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" Nov 03 20:25:28.999449 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:28.999149 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xsp2\" (UniqueName: \"kubernetes.io/projected/031f50c1-6a58-454b-a132-2291aa316cd3-kube-api-access-7xsp2\") pod \"maas-api-revoke-sub-461f444c-pz656\" (UID: \"031f50c1-6a58-454b-a132-2291aa316cd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" Nov 03 20:25:29.151415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:29.139500 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" Nov 03 20:25:29.654006 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:25:29.653976 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031f50c1_6a58_454b_a132_2291aa316cd3.slice/crio-0f0eefd76b0c9a631802841c35005395fac18fa380e64807133994bd6781c974 WatchSource:0}: Error finding container 0f0eefd76b0c9a631802841c35005395fac18fa380e64807133994bd6781c974: Status 404 returned error can't find the container with id 0f0eefd76b0c9a631802841c35005395fac18fa380e64807133994bd6781c974 Nov 03 20:25:29.655158 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:29.655129 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656"] Nov 03 20:25:30.085360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:30.085262 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" event={"ID":"031f50c1-6a58-454b-a132-2291aa316cd3","Type":"ContainerStarted","Data":"0f0eefd76b0c9a631802841c35005395fac18fa380e64807133994bd6781c974"} Nov 03 20:25:30.902071 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:30.902042 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" Nov 03 20:25:31.026848 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:31.026812 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf"] Nov 03 20:25:31.034550 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:31.034516 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d278d729-2mdhf"] Nov 03 20:25:31.098171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:31.098135 4457 generic.go:356] "Generic (PLEG): container finished" podID="031f50c1-6a58-454b-a132-2291aa316cd3" containerID="978e42784495c709eec455d0195d98fdebfa842d900fef834c07d84a557d74f9" exitCode=0 Nov 03 20:25:31.098645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:31.098215 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" event={"ID":"031f50c1-6a58-454b-a132-2291aa316cd3","Type":"ContainerDied","Data":"978e42784495c709eec455d0195d98fdebfa842d900fef834c07d84a557d74f9"} Nov 03 20:25:31.778229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:31.778187 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ab47a3-5965-48cb-b127-4c927ceaf51a" path="/var/lib/kubelet/pods/c6ab47a3-5965-48cb-b127-4c927ceaf51a/volumes" Nov 03 20:25:32.365811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:32.365752 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" Nov 03 20:25:32.407150 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:32.407117 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031f50c1-6a58-454b-a132-2291aa316cd3-kube-api-access-7xsp2\" (UniqueName: \"kubernetes.io/projected/031f50c1-6a58-454b-a132-2291aa316cd3-kube-api-access-7xsp2\") pod \"031f50c1-6a58-454b-a132-2291aa316cd3\" (UID: \"031f50c1-6a58-454b-a132-2291aa316cd3\") " Nov 03 20:25:32.407319 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:32.407260 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-xks-ca\") pod \"031f50c1-6a58-454b-a132-2291aa316cd3\" (UID: \"031f50c1-6a58-454b-a132-2291aa316cd3\") " Nov 03 20:25:32.407429 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:32.407380 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-openshift-ca\") pod \"031f50c1-6a58-454b-a132-2291aa316cd3\" (UID: \"031f50c1-6a58-454b-a132-2291aa316cd3\") " Nov 03 20:25:32.407843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:32.407815 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-openshift-ca" pod "031f50c1-6a58-454b-a132-2291aa316cd3" (UID: "031f50c1-6a58-454b-a132-2291aa316cd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:32.408711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:32.408439 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:32.409626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:32.409602 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-xks-ca" pod "031f50c1-6a58-454b-a132-2291aa316cd3" (UID: "031f50c1-6a58-454b-a132-2291aa316cd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:32.410930 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:32.410905 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031f50c1-6a58-454b-a132-2291aa316cd3-kube-api-access-7xsp2" pod "031f50c1-6a58-454b-a132-2291aa316cd3" (UID: "031f50c1-6a58-454b-a132-2291aa316cd3"). InnerVolumeSpecName "kube-api-access-7xsp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:32.511416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:32.511118 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xsp2\" (UniqueName: \"kubernetes.io/projected/031f50c1-6a58-454b-a132-2291aa316cd3-kube-api-access-7xsp2\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:32.511416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:32.511174 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/031f50c1-6a58-454b-a132-2291aa316cd3-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:33.109050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.109014 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk"] Nov 03 20:25:33.109980 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.109957 4457 cpu_manager.go:414] "containerMap: removing container" podUID="031f50c1-6a58-454b-a132-2291aa316cd3" containerName="revoke-keys" Nov 03 20:25:33.109980 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.109979 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031f50c1-6a58-454b-a132-2291aa316cd3" containerName="revoke-keys" Nov 03 20:25:33.110163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.110152 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="031f50c1-6a58-454b-a132-2291aa316cd3" containerName="revoke-keys" Nov 03 20:25:33.121322 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.119629 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" Nov 03 20:25:33.137573 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.132357 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk"] Nov 03 20:25:33.147411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.145434 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" event={"ID":"031f50c1-6a58-454b-a132-2291aa316cd3","Type":"ContainerDied","Data":"0f0eefd76b0c9a631802841c35005395fac18fa380e64807133994bd6781c974"} Nov 03 20:25:33.147411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.145503 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f0eefd76b0c9a631802841c35005395fac18fa380e64807133994bd6781c974" Nov 03 20:25:33.147411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.145592 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656" Nov 03 20:25:33.166958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.166906 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx6kz\" (UniqueName: \"kubernetes.io/projected/592b66d4-4ef3-4941-b576-657982ac082e-kube-api-access-dx6kz\") pod \"maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk\" (UID: \"592b66d4-4ef3-4941-b576-657982ac082e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" Nov 03 20:25:33.167141 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.166982 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk\" (UID: \"592b66d4-4ef3-4941-b576-657982ac082e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" Nov 03 20:25:33.170437 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.167711 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk\" (UID: \"592b66d4-4ef3-4941-b576-657982ac082e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" Nov 03 20:25:33.269418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.269176 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk\" (UID: \"592b66d4-4ef3-4941-b576-657982ac082e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" Nov 03 20:25:33.269606 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.269500 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx6kz\" (UniqueName: \"kubernetes.io/projected/592b66d4-4ef3-4941-b576-657982ac082e-kube-api-access-dx6kz\") pod \"maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk\" (UID: \"592b66d4-4ef3-4941-b576-657982ac082e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" Nov 03 20:25:33.269606 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.269570 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk\" (UID: \"592b66d4-4ef3-4941-b576-657982ac082e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" Nov 03 20:25:33.271597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.270498 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk\" (UID: \"592b66d4-4ef3-4941-b576-657982ac082e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" Nov 03 20:25:33.275808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.275783 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk\" (UID: \"592b66d4-4ef3-4941-b576-657982ac082e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" Nov 03 20:25:33.282752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.282728 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx6kz\" (UniqueName: \"kubernetes.io/projected/592b66d4-4ef3-4941-b576-657982ac082e-kube-api-access-dx6kz\") pod \"maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk\" (UID: \"592b66d4-4ef3-4941-b576-657982ac082e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" Nov 03 20:25:33.365565 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.365484 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s"] Nov 03 20:25:33.365712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.365684 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" containerID="cri-o://d636455cacf019a44c5dfccedba6a25e38f79387f31811a7af23cba6f12c6360" gracePeriod=30 Nov 03 20:25:33.497037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.496997 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" Nov 03 20:25:33.796379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:33.796351 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk"] Nov 03 20:25:33.799185 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:25:33.799153 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592b66d4_4ef3_4941_b576_657982ac082e.slice/crio-04d8762b2efa008a333fd5d7f172d02778cb0f2bf5f97527e671fe1d9f6cdc91 WatchSource:0}: Error finding container 04d8762b2efa008a333fd5d7f172d02778cb0f2bf5f97527e671fe1d9f6cdc91: Status 404 returned error can't find the container with id 04d8762b2efa008a333fd5d7f172d02778cb0f2bf5f97527e671fe1d9f6cdc91 Nov 03 20:25:34.157582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:34.157540 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" event={"ID":"592b66d4-4ef3-4941-b576-657982ac082e","Type":"ContainerStarted","Data":"04d8762b2efa008a333fd5d7f172d02778cb0f2bf5f97527e671fe1d9f6cdc91"} Nov 03 20:25:35.168825 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:35.168788 4457 generic.go:356] "Generic (PLEG): container finished" podID="592b66d4-4ef3-4941-b576-657982ac082e" containerID="21367622f69c2b9da0e2954ed77bb4886acaf7f66c98cbbb6317b5b219e61943" exitCode=0 Nov 03 20:25:35.169223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:35.168883 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" event={"ID":"592b66d4-4ef3-4941-b576-657982ac082e","Type":"ContainerDied","Data":"21367622f69c2b9da0e2954ed77bb4886acaf7f66c98cbbb6317b5b219e61943"} Nov 03 20:25:36.149444 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:25:36.149376 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:25:36.368338 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:36.368310 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" Nov 03 20:25:36.434748 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:36.434669 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-openshift-ca\") pod \"592b66d4-4ef3-4941-b576-657982ac082e\" (UID: \"592b66d4-4ef3-4941-b576-657982ac082e\") " Nov 03 20:25:36.434993 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:36.434827 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/592b66d4-4ef3-4941-b576-657982ac082e-kube-api-access-dx6kz\" (UniqueName: \"kubernetes.io/projected/592b66d4-4ef3-4941-b576-657982ac082e-kube-api-access-dx6kz\") pod \"592b66d4-4ef3-4941-b576-657982ac082e\" (UID: \"592b66d4-4ef3-4941-b576-657982ac082e\") " Nov 03 20:25:36.434993 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:36.434934 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-xks-ca\") pod \"592b66d4-4ef3-4941-b576-657982ac082e\" (UID: \"592b66d4-4ef3-4941-b576-657982ac082e\") " Nov 03 20:25:36.435142 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:36.435100 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-openshift-ca" pod "592b66d4-4ef3-4941-b576-657982ac082e" (UID: "592b66d4-4ef3-4941-b576-657982ac082e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:36.435985 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:36.435958 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:36.437471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:36.437441 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-xks-ca" pod "592b66d4-4ef3-4941-b576-657982ac082e" (UID: "592b66d4-4ef3-4941-b576-657982ac082e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:25:36.437586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:36.437563 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/592b66d4-4ef3-4941-b576-657982ac082e-kube-api-access-dx6kz" pod "592b66d4-4ef3-4941-b576-657982ac082e" (UID: "592b66d4-4ef3-4941-b576-657982ac082e"). InnerVolumeSpecName "kube-api-access-dx6kz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:36.551519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:36.551476 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx6kz\" (UniqueName: \"kubernetes.io/projected/592b66d4-4ef3-4941-b576-657982ac082e-kube-api-access-dx6kz\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:36.551519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:36.551520 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/592b66d4-4ef3-4941-b576-657982ac082e-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:37.191365 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:37.191336 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" Nov 03 20:25:37.191609 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:37.191332 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk" event={"ID":"592b66d4-4ef3-4941-b576-657982ac082e","Type":"ContainerDied","Data":"04d8762b2efa008a333fd5d7f172d02778cb0f2bf5f97527e671fe1d9f6cdc91"} Nov 03 20:25:37.191609 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:37.191483 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04d8762b2efa008a333fd5d7f172d02778cb0f2bf5f97527e671fe1d9f6cdc91" Nov 03 20:25:38.750165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.750137 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:25:38.832403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.832352 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-data\") pod \"3062e080-8433-4f4b-b0e3-ffc0da402d66\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " Nov 03 20:25:38.832601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.832488 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-envoy\") pod \"3062e080-8433-4f4b-b0e3-ffc0da402d66\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " Nov 03 20:25:38.832601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.832561 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3062e080-8433-4f4b-b0e3-ffc0da402d66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3062e080-8433-4f4b-b0e3-ffc0da402d66-istiod-ca-cert\") pod \"3062e080-8433-4f4b-b0e3-ffc0da402d66\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " Nov 03 20:25:38.832738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.832659 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-token\" (UniqueName: \"kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-token\") pod \"3062e080-8433-4f4b-b0e3-ffc0da402d66\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " Nov 03 20:25:38.832738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.832720 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-kube-api-access-9bjwt\" (UniqueName: \"kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-kube-api-access-9bjwt\") pod \"3062e080-8433-4f4b-b0e3-ffc0da402d66\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " Nov 03 20:25:38.832844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.832740 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-data" pod "3062e080-8433-4f4b-b0e3-ffc0da402d66" (UID: "3062e080-8433-4f4b-b0e3-ffc0da402d66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.832844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.832755 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-certs\") pod \"3062e080-8433-4f4b-b0e3-ffc0da402d66\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " Nov 03 20:25:38.832958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.832884 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-credential-socket\") pod \"3062e080-8433-4f4b-b0e3-ffc0da402d66\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " Nov 03 20:25:38.833015 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.832997 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-certs" pod "3062e080-8433-4f4b-b0e3-ffc0da402d66" (UID: "3062e080-8433-4f4b-b0e3-ffc0da402d66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.833079 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.833023 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3062e080-8433-4f4b-b0e3-ffc0da402d66-istiod-ca-cert" pod "3062e080-8433-4f4b-b0e3-ffc0da402d66" (UID: "3062e080-8433-4f4b-b0e3-ffc0da402d66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:38.833079 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.833058 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-socket\") pod \"3062e080-8433-4f4b-b0e3-ffc0da402d66\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " Nov 03 20:25:38.833192 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.833097 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-podinfo\") pod \"3062e080-8433-4f4b-b0e3-ffc0da402d66\" (UID: \"3062e080-8433-4f4b-b0e3-ffc0da402d66\") " Nov 03 20:25:38.833360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.833337 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-credential-socket" pod "3062e080-8433-4f4b-b0e3-ffc0da402d66" (UID: "3062e080-8433-4f4b-b0e3-ffc0da402d66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.833573 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.833546 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-socket" pod "3062e080-8433-4f4b-b0e3-ffc0da402d66" (UID: "3062e080-8433-4f4b-b0e3-ffc0da402d66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.834257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.834228 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.834356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.834263 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.834356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.834279 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.834356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.834292 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3062e080-8433-4f4b-b0e3-ffc0da402d66-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.834356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.834306 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.836513 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.836482 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-envoy" pod "3062e080-8433-4f4b-b0e3-ffc0da402d66" (UID: "3062e080-8433-4f4b-b0e3-ffc0da402d66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:25:38.836513 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.836495 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-kube-api-access-9bjwt" pod "3062e080-8433-4f4b-b0e3-ffc0da402d66" (UID: "3062e080-8433-4f4b-b0e3-ffc0da402d66"). InnerVolumeSpecName "kube-api-access-9bjwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:38.836673 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.836554 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-token" pod "3062e080-8433-4f4b-b0e3-ffc0da402d66" (UID: "3062e080-8433-4f4b-b0e3-ffc0da402d66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:38.836673 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.836621 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-podinfo" pod "3062e080-8433-4f4b-b0e3-ffc0da402d66" (UID: "3062e080-8433-4f4b-b0e3-ffc0da402d66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:25:38.945816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.945763 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.945816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.945806 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.945816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.945821 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:38.946153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:38.945836 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bjwt\" (UniqueName: \"kubernetes.io/projected/3062e080-8433-4f4b-b0e3-ffc0da402d66-kube-api-access-9bjwt\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:39.216785 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:39.216746 4457 generic.go:356] "Generic (PLEG): container finished" podID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerID="d636455cacf019a44c5dfccedba6a25e38f79387f31811a7af23cba6f12c6360" exitCode=0 Nov 03 20:25:39.216973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:39.216792 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" Nov 03 20:25:39.216973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:39.216836 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" event={"ID":"3062e080-8433-4f4b-b0e3-ffc0da402d66","Type":"ContainerDied","Data":"d636455cacf019a44c5dfccedba6a25e38f79387f31811a7af23cba6f12c6360"} Nov 03 20:25:39.216973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:39.216900 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s" event={"ID":"3062e080-8433-4f4b-b0e3-ffc0da402d66","Type":"ContainerDied","Data":"363e62a2709004a80e985df14f14236f27890e9561aacded12f65f57905910eb"} Nov 03 20:25:39.216973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:39.216926 4457 scope.go:122] "RemoveContainer" containerID="d636455cacf019a44c5dfccedba6a25e38f79387f31811a7af23cba6f12c6360" Nov 03 20:25:39.230455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:39.230435 4457 scope.go:122] "RemoveContainer" containerID="d636455cacf019a44c5dfccedba6a25e38f79387f31811a7af23cba6f12c6360" Nov 03 20:25:39.230725 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:25:39.230705 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d636455cacf019a44c5dfccedba6a25e38f79387f31811a7af23cba6f12c6360\": container with ID starting with d636455cacf019a44c5dfccedba6a25e38f79387f31811a7af23cba6f12c6360 not found: ID does not exist" containerID="d636455cacf019a44c5dfccedba6a25e38f79387f31811a7af23cba6f12c6360" Nov 03 20:25:39.230782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:39.230733 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d636455cacf019a44c5dfccedba6a25e38f79387f31811a7af23cba6f12c6360"} err="failed to get container status \"d636455cacf019a44c5dfccedba6a25e38f79387f31811a7af23cba6f12c6360\": rpc error: code = NotFound desc = could not find container \"d636455cacf019a44c5dfccedba6a25e38f79387f31811a7af23cba6f12c6360\": container with ID starting with d636455cacf019a44c5dfccedba6a25e38f79387f31811a7af23cba6f12c6360 not found: ID does not exist" Nov 03 20:25:39.247231 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:39.247201 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s"] Nov 03 20:25:39.252729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:39.252702 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-5pb9s"] Nov 03 20:25:39.778495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:39.778328 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" path="/var/lib/kubelet/pods/3062e080-8433-4f4b-b0e3-ffc0da402d66/volumes" Nov 03 20:25:43.234680 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:43.234640 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs"] Nov 03 20:25:43.252473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:43.252438 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl"] Nov 03 20:25:45.394025 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.393993 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv"] Nov 03 20:25:45.395030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.395009 4457 cpu_manager.go:414] "containerMap: removing container" podUID="592b66d4-4ef3-4941-b576-657982ac082e" containerName="revoke-keys" Nov 03 20:25:45.395093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.395031 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="592b66d4-4ef3-4941-b576-657982ac082e" containerName="revoke-keys" Nov 03 20:25:45.395093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.395058 4457 cpu_manager.go:414] "containerMap: removing container" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" Nov 03 20:25:45.395093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.395063 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" Nov 03 20:25:45.395263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.395190 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="592b66d4-4ef3-4941-b576-657982ac082e" containerName="revoke-keys" Nov 03 20:25:45.395263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.395210 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="3062e080-8433-4f4b-b0e3-ffc0da402d66" containerName="istio-proxy" Nov 03 20:25:45.400809 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.400790 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" Nov 03 20:25:45.411025 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.410994 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv"] Nov 03 20:25:45.411639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.411612 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c2c601\"" Nov 03 20:25:45.441466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.441431 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g"] Nov 03 20:25:45.447076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.447055 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" Nov 03 20:25:45.480897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.480861 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c2c601-dockercfg-pmt9s\"" Nov 03 20:25:45.492134 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.492100 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4987\" (UniqueName: \"kubernetes.io/projected/896c36e5-f4a0-479e-b3c9-7f16881b272e-kube-api-access-n4987\") pod \"payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g\" (UID: \"896c36e5-f4a0-479e-b3c9-7f16881b272e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" Nov 03 20:25:45.492375 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.492356 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv\" (UID: \"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" Nov 03 20:25:45.492988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.492653 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896c36e5-f4a0-479e-b3c9-7f16881b272e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g\" (UID: \"896c36e5-f4a0-479e-b3c9-7f16881b272e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" Nov 03 20:25:45.493236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.493212 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbqz\" (UniqueName: \"kubernetes.io/projected/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-kube-api-access-zsbqz\") pod \"payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv\" (UID: \"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" Nov 03 20:25:45.508865 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.508828 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g"] Nov 03 20:25:45.708487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.703016 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv\" (UID: \"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" Nov 03 20:25:45.708487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.703105 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896c36e5-f4a0-479e-b3c9-7f16881b272e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g\" (UID: \"896c36e5-f4a0-479e-b3c9-7f16881b272e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" Nov 03 20:25:45.708487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.703214 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbqz\" (UniqueName: \"kubernetes.io/projected/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-kube-api-access-zsbqz\") pod \"payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv\" (UID: \"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" Nov 03 20:25:45.708487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.703497 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4987\" (UniqueName: \"kubernetes.io/projected/896c36e5-f4a0-479e-b3c9-7f16881b272e-kube-api-access-n4987\") pod \"payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g\" (UID: \"896c36e5-f4a0-479e-b3c9-7f16881b272e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" Nov 03 20:25:45.708487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.707012 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896c36e5-f4a0-479e-b3c9-7f16881b272e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g\" (UID: \"896c36e5-f4a0-479e-b3c9-7f16881b272e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" Nov 03 20:25:45.708487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.707189 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv\" (UID: \"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" Nov 03 20:25:45.744147 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.744111 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4987\" (UniqueName: \"kubernetes.io/projected/896c36e5-f4a0-479e-b3c9-7f16881b272e-kube-api-access-n4987\") pod \"payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g\" (UID: \"896c36e5-f4a0-479e-b3c9-7f16881b272e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" Nov 03 20:25:45.746114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.746087 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbqz\" (UniqueName: \"kubernetes.io/projected/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-kube-api-access-zsbqz\") pod \"payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv\" (UID: \"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" Nov 03 20:25:45.802758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.802722 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" Nov 03 20:25:45.869454 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.869019 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk"] Nov 03 20:25:45.877133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.877104 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:45.882805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.882772 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk"] Nov 03 20:25:45.883180 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.883158 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-34e844-openshift-default-dockercfg-k295l\"" Nov 03 20:25:45.941419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.941369 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-certs\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:45.941779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.941754 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istiod-ca-cert\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:45.942030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.942010 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-data\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:45.942331 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.942313 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-envoy\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:45.942540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.942524 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-podinfo\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:45.942708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.942692 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-token\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:45.948805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.948776 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-credential-socket\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:45.949187 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.949154 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-socket\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:45.949544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:45.949442 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9fkm\" (UniqueName: \"kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-kube-api-access-f9fkm\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.039549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.033873 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" Nov 03 20:25:46.073650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.069346 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-envoy\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.073650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.069452 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-podinfo\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.073650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.069508 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-token\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.073650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.069570 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-credential-socket\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.073650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.069683 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-socket\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.073650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.069729 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9fkm\" (UniqueName: \"kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-kube-api-access-f9fkm\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.073650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.069881 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-certs\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.073650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.069926 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istiod-ca-cert\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.073650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.069985 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-data\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.073650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.070902 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-credential-socket\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.073650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.073086 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-socket\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.073650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.073150 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-certs\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.084128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.074762 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istiod-ca-cert\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.084128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.075264 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-data\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.098264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.095331 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-token\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.103545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.101608 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-podinfo\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.105006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.104568 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-envoy\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.120436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.106223 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9fkm\" (UniqueName: \"kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-kube-api-access-f9fkm\") pod \"e2e-inf-b-34e844-openshift-default-7775789977-gqrdk\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.232457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.232093 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g"] Nov 03 20:25:46.234470 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:25:46.234409 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896c36e5_f4a0_479e_b3c9_7f16881b272e.slice/crio-31327884ecf7688d79db048aa3b47bb8f8b2588437cde08b0be4002c20f55514 WatchSource:0}: Error finding container 31327884ecf7688d79db048aa3b47bb8f8b2588437cde08b0be4002c20f55514: Status 404 returned error can't find the container with id 31327884ecf7688d79db048aa3b47bb8f8b2588437cde08b0be4002c20f55514 Nov 03 20:25:46.340132 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.340102 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:46.410275 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:25:46.409212 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e677dc_c502_43a4_9bdf_d7983a6f9b1b.slice/crio-a36fa267e1bbcf74c80c82c863c8c179e7c1fd4523f6bc9a23904f76dbfc70d8 WatchSource:0}: Error finding container a36fa267e1bbcf74c80c82c863c8c179e7c1fd4523f6bc9a23904f76dbfc70d8: Status 404 returned error can't find the container with id a36fa267e1bbcf74c80c82c863c8c179e7c1fd4523f6bc9a23904f76dbfc70d8 Nov 03 20:25:46.420855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.420827 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv"] Nov 03 20:25:46.641302 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:25:46.641266 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117fccf6_6e71_49c1_a9d0_d5f0c4d6eb22.slice/crio-94eae0847f62f13493081e5726f28f8dae6c4fac43b4a6bf5df57ce12fe098a3 WatchSource:0}: Error finding container 94eae0847f62f13493081e5726f28f8dae6c4fac43b4a6bf5df57ce12fe098a3: Status 404 returned error can't find the container with id 94eae0847f62f13493081e5726f28f8dae6c4fac43b4a6bf5df57ce12fe098a3 Nov 03 20:25:46.642786 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.642760 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk"] Nov 03 20:25:46.644590 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.644553 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:46.644693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.644637 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:46.644763 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:46.644703 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:47.052121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.052064 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" event={"ID":"896c36e5-f4a0-479e-b3c9-7f16881b272e","Type":"ContainerStarted","Data":"7e7d3c376499619d5e61271a98b04d9a2234399edaf0e6f233ad1b214c5e1255"} Nov 03 20:25:47.052353 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.052154 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" event={"ID":"896c36e5-f4a0-479e-b3c9-7f16881b272e","Type":"ContainerStarted","Data":"31327884ecf7688d79db048aa3b47bb8f8b2588437cde08b0be4002c20f55514"} Nov 03 20:25:47.054179 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.053538 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" Nov 03 20:25:47.060431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.055995 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" event={"ID":"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22","Type":"ContainerStarted","Data":"4dd34ca74dc2ca470921cdaaf15561ce5bbaebac9ff00968b6b051ec23e115eb"} Nov 03 20:25:47.060431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.056041 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" event={"ID":"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22","Type":"ContainerStarted","Data":"94eae0847f62f13493081e5726f28f8dae6c4fac43b4a6bf5df57ce12fe098a3"} Nov 03 20:25:47.060431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.059618 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" event={"ID":"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b","Type":"ContainerStarted","Data":"eb792ba6fff2a1dc5eca7f82dc1ae3c95d97351e7dcc6e23f3da0898f4bd1818"} Nov 03 20:25:47.060431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.059714 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" Nov 03 20:25:47.060431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.059743 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" event={"ID":"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b","Type":"ContainerStarted","Data":"a36fa267e1bbcf74c80c82c863c8c179e7c1fd4523f6bc9a23904f76dbfc70d8"} Nov 03 20:25:47.077442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.077344 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" podStartSLOduration=2.077327799 podStartE2EDuration="2.077327799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:47.072898835 +0000 UTC m=+1725.448495609" watchObservedRunningTime="2026-11-03 20:25:47.077327799 +0000 UTC m=+1725.452924547" Nov 03 20:25:47.096364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.096308 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" podStartSLOduration=2.096291449 podStartE2EDuration="2.096291449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:47.092510241 +0000 UTC m=+1725.468106989" watchObservedRunningTime="2026-11-03 20:25:47.096291449 +0000 UTC m=+1725.471888230" Nov 03 20:25:47.127593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.127528 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" podStartSLOduration=2.127505701 podStartE2EDuration="2.127505701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:47.120053392 +0000 UTC m=+1725.495650139" watchObservedRunningTime="2026-11-03 20:25:47.127505701 +0000 UTC m=+1725.503102448" Nov 03 20:25:47.340882 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.340772 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:47.343506 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.343473 4457 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:25:47.343717 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:47.343540 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" podUID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 03 20:25:48.235536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:48.235487 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" podUID="73bfef0b-3309-464e-97ed-2eec4aa76d9b" containerName="payload-processing" containerID="cri-o://b5936e827bb7f6d742901718e5a55c88c45b1b370ae13c5975c7faeb7333d30b" gracePeriod=25 Nov 03 20:25:48.253076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:48.253033 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" podUID="14547b1f-02fe-43a7-b6c1-c2165dac0f17" containerName="payload-pre-processing" containerID="cri-o://8492a32a4faf34988dbac18f4358548536b195483a4fa8f2c85d7cc014c70297" gracePeriod=25 Nov 03 20:25:48.341007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:48.340967 4457 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:25:48.341163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:48.341041 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" podUID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 03 20:25:48.805857 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:48.805820 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" Nov 03 20:25:48.847868 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:48.847833 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73bfef0b-3309-464e-97ed-2eec4aa76d9b-kube-api-access-5t587\" (UniqueName: \"kubernetes.io/projected/73bfef0b-3309-464e-97ed-2eec4aa76d9b-kube-api-access-5t587\") pod \"73bfef0b-3309-464e-97ed-2eec4aa76d9b\" (UID: \"73bfef0b-3309-464e-97ed-2eec4aa76d9b\") " Nov 03 20:25:48.848188 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:48.848169 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73bfef0b-3309-464e-97ed-2eec4aa76d9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bfef0b-3309-464e-97ed-2eec4aa76d9b-plugins-config-volume\") pod \"73bfef0b-3309-464e-97ed-2eec4aa76d9b\" (UID: \"73bfef0b-3309-464e-97ed-2eec4aa76d9b\") " Nov 03 20:25:48.848720 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:48.848687 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73bfef0b-3309-464e-97ed-2eec4aa76d9b-plugins-config-volume" pod "73bfef0b-3309-464e-97ed-2eec4aa76d9b" (UID: "73bfef0b-3309-464e-97ed-2eec4aa76d9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:48.849905 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:48.849107 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73bfef0b-3309-464e-97ed-2eec4aa76d9b-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:48.853915 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:48.853888 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73bfef0b-3309-464e-97ed-2eec4aa76d9b-kube-api-access-5t587" pod "73bfef0b-3309-464e-97ed-2eec4aa76d9b" (UID: "73bfef0b-3309-464e-97ed-2eec4aa76d9b"). InnerVolumeSpecName "kube-api-access-5t587". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:48.929186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:48.929155 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" Nov 03 20:25:48.951435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:48.951376 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t587\" (UniqueName: \"kubernetes.io/projected/73bfef0b-3309-464e-97ed-2eec4aa76d9b-kube-api-access-5t587\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:49.052745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.052715 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14547b1f-02fe-43a7-b6c1-c2165dac0f17-kube-api-access-gcvxd\" (UniqueName: \"kubernetes.io/projected/14547b1f-02fe-43a7-b6c1-c2165dac0f17-kube-api-access-gcvxd\") pod \"14547b1f-02fe-43a7-b6c1-c2165dac0f17\" (UID: \"14547b1f-02fe-43a7-b6c1-c2165dac0f17\") " Nov 03 20:25:49.052955 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.052943 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14547b1f-02fe-43a7-b6c1-c2165dac0f17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14547b1f-02fe-43a7-b6c1-c2165dac0f17-plugins-config-volume\") pod \"14547b1f-02fe-43a7-b6c1-c2165dac0f17\" (UID: \"14547b1f-02fe-43a7-b6c1-c2165dac0f17\") " Nov 03 20:25:49.053352 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.053318 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14547b1f-02fe-43a7-b6c1-c2165dac0f17-plugins-config-volume" pod "14547b1f-02fe-43a7-b6c1-c2165dac0f17" (UID: "14547b1f-02fe-43a7-b6c1-c2165dac0f17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:25:49.053506 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.053490 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14547b1f-02fe-43a7-b6c1-c2165dac0f17-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:49.054840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.054822 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14547b1f-02fe-43a7-b6c1-c2165dac0f17-kube-api-access-gcvxd" pod "14547b1f-02fe-43a7-b6c1-c2165dac0f17" (UID: "14547b1f-02fe-43a7-b6c1-c2165dac0f17"). InnerVolumeSpecName "kube-api-access-gcvxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:25:49.128441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.126515 4457 generic.go:356] "Generic (PLEG): container finished" podID="73bfef0b-3309-464e-97ed-2eec4aa76d9b" containerID="b5936e827bb7f6d742901718e5a55c88c45b1b370ae13c5975c7faeb7333d30b" exitCode=0 Nov 03 20:25:49.128441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.126561 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" Nov 03 20:25:49.128441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.126607 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" event={"ID":"73bfef0b-3309-464e-97ed-2eec4aa76d9b","Type":"ContainerDied","Data":"b5936e827bb7f6d742901718e5a55c88c45b1b370ae13c5975c7faeb7333d30b"} Nov 03 20:25:49.128441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.126651 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs" event={"ID":"73bfef0b-3309-464e-97ed-2eec4aa76d9b","Type":"ContainerDied","Data":"af95295d144fef92112fc73dae8e70bc61bf58bc3d10eccd3886a46df87d34d5"} Nov 03 20:25:49.128441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.126673 4457 scope.go:122] "RemoveContainer" containerID="b5936e827bb7f6d742901718e5a55c88c45b1b370ae13c5975c7faeb7333d30b" Nov 03 20:25:49.131112 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.131088 4457 generic.go:356] "Generic (PLEG): container finished" podID="14547b1f-02fe-43a7-b6c1-c2165dac0f17" containerID="8492a32a4faf34988dbac18f4358548536b195483a4fa8f2c85d7cc014c70297" exitCode=0 Nov 03 20:25:49.133191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.131149 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" Nov 03 20:25:49.133191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.131168 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" event={"ID":"14547b1f-02fe-43a7-b6c1-c2165dac0f17","Type":"ContainerDied","Data":"8492a32a4faf34988dbac18f4358548536b195483a4fa8f2c85d7cc014c70297"} Nov 03 20:25:49.133191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.131484 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" event={"ID":"14547b1f-02fe-43a7-b6c1-c2165dac0f17","Type":"ContainerDied","Data":"bc9d36c401dacb2f06c1d35fc9787aab378108f629c720bafb838f401e04396f"} Nov 03 20:25:49.144169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.144130 4457 scope.go:122] "RemoveContainer" containerID="b5936e827bb7f6d742901718e5a55c88c45b1b370ae13c5975c7faeb7333d30b" Nov 03 20:25:49.144529 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:25:49.144504 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5936e827bb7f6d742901718e5a55c88c45b1b370ae13c5975c7faeb7333d30b\": container with ID starting with b5936e827bb7f6d742901718e5a55c88c45b1b370ae13c5975c7faeb7333d30b not found: ID does not exist" containerID="b5936e827bb7f6d742901718e5a55c88c45b1b370ae13c5975c7faeb7333d30b" Nov 03 20:25:49.144620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.144535 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5936e827bb7f6d742901718e5a55c88c45b1b370ae13c5975c7faeb7333d30b"} err="failed to get container status \"b5936e827bb7f6d742901718e5a55c88c45b1b370ae13c5975c7faeb7333d30b\": rpc error: code = NotFound desc = could not find container \"b5936e827bb7f6d742901718e5a55c88c45b1b370ae13c5975c7faeb7333d30b\": container with ID starting with b5936e827bb7f6d742901718e5a55c88c45b1b370ae13c5975c7faeb7333d30b not found: ID does not exist" Nov 03 20:25:49.144620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.144554 4457 scope.go:122] "RemoveContainer" containerID="8492a32a4faf34988dbac18f4358548536b195483a4fa8f2c85d7cc014c70297" Nov 03 20:25:49.155441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.155378 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcvxd\" (UniqueName: \"kubernetes.io/projected/14547b1f-02fe-43a7-b6c1-c2165dac0f17-kube-api-access-gcvxd\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:25:49.172504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.172427 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs"] Nov 03 20:25:49.181842 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.173002 4457 scope.go:122] "RemoveContainer" containerID="8492a32a4faf34988dbac18f4358548536b195483a4fa8f2c85d7cc014c70297" Nov 03 20:25:49.181842 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:25:49.173376 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8492a32a4faf34988dbac18f4358548536b195483a4fa8f2c85d7cc014c70297\": container with ID starting with 8492a32a4faf34988dbac18f4358548536b195483a4fa8f2c85d7cc014c70297 not found: ID does not exist" containerID="8492a32a4faf34988dbac18f4358548536b195483a4fa8f2c85d7cc014c70297" Nov 03 20:25:49.181842 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.173427 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8492a32a4faf34988dbac18f4358548536b195483a4fa8f2c85d7cc014c70297"} err="failed to get container status \"8492a32a4faf34988dbac18f4358548536b195483a4fa8f2c85d7cc014c70297\": rpc error: code = NotFound desc = could not find container \"8492a32a4faf34988dbac18f4358548536b195483a4fa8f2c85d7cc014c70297\": container with ID starting with 8492a32a4faf34988dbac18f4358548536b195483a4fa8f2c85d7cc014c70297 not found: ID does not exist" Nov 03 20:25:49.200427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.199496 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f3ae23-764ff584c6-cxcfs"] Nov 03 20:25:49.205947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.205918 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl"] Nov 03 20:25:49.215001 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.214931 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl"] Nov 03 20:25:49.341107 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.341033 4457 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:25:49.341107 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.341089 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" podUID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 03 20:25:49.790470 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.790131 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14547b1f-02fe-43a7-b6c1-c2165dac0f17" path="/var/lib/kubelet/pods/14547b1f-02fe-43a7-b6c1-c2165dac0f17/volumes" Nov 03 20:25:49.792482 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.790896 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73bfef0b-3309-464e-97ed-2eec4aa76d9b" path="/var/lib/kubelet/pods/73bfef0b-3309-464e-97ed-2eec4aa76d9b/volumes" Nov 03 20:25:49.798514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:49.798468 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f3ae23-778d5b7d9f-7mshl" podUID="14547b1f-02fe-43a7-b6c1-c2165dac0f17" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.171:50052\" within 1s: context deadline exceeded" Nov 03 20:25:49.798849 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:25:49.798827 4457 logging.go:55] [core] [Channel #3005 SubChannel #3006]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.171:50052", ServerName: "10.133.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.171:50052: operation was canceled" Nov 03 20:25:50.341578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:50.341542 4457 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:25:50.342119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:50.341599 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" podUID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 03 20:25:51.341436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:51.341359 4457 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:25:51.341623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:51.341441 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" podUID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 03 20:25:52.341319 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.341281 4457 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:25:52.341581 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.341344 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" podUID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 03 20:25:52.629057 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.628963 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz"] Nov 03 20:25:52.630109 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.630084 4457 cpu_manager.go:414] "containerMap: removing container" podUID="14547b1f-02fe-43a7-b6c1-c2165dac0f17" containerName="payload-pre-processing" Nov 03 20:25:52.630109 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.630108 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14547b1f-02fe-43a7-b6c1-c2165dac0f17" containerName="payload-pre-processing" Nov 03 20:25:52.630288 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.630199 4457 cpu_manager.go:414] "containerMap: removing container" podUID="73bfef0b-3309-464e-97ed-2eec4aa76d9b" containerName="payload-processing" Nov 03 20:25:52.630288 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.630208 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73bfef0b-3309-464e-97ed-2eec4aa76d9b" containerName="payload-processing" Nov 03 20:25:52.630462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.630446 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="14547b1f-02fe-43a7-b6c1-c2165dac0f17" containerName="payload-pre-processing" Nov 03 20:25:52.630525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.630485 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="73bfef0b-3309-464e-97ed-2eec4aa76d9b" containerName="payload-processing" Nov 03 20:25:52.635877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.635857 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.643360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.643338 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5de824-openshift-default-dockercfg-g4q4k\"" Nov 03 20:25:52.653047 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.653020 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz"] Nov 03 20:25:52.694507 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.694468 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istiod-ca-cert\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.694712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.694559 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-data\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.694712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.694647 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-socket\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.694712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.694705 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-envoy\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.694891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.694814 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-credential-socket\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.694891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.694839 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-certs\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.694891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.694879 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-token\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.695052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.695027 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-podinfo\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.695108 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.695065 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5trk\" (UniqueName: \"kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-kube-api-access-l5trk\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.796706 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-socket\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.796774 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-envoy\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.796844 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-credential-socket\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.796858 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-certs\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.796883 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-token\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.797466 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-podinfo\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.797517 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5trk\" (UniqueName: \"kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-kube-api-access-l5trk\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.797711 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istiod-ca-cert\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.797806 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-data\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.798428 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-socket\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.798816 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-data\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.799178 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-credential-socket\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.799520 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-certs\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.799721 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-envoy\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.801332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.800696 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istiod-ca-cert\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.802926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.802726 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-podinfo\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.810552 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.810527 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-token\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.810773 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.810751 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5trk\" (UniqueName: \"kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-kube-api-access-l5trk\") pod \"e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:52.961469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:52.959669 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:53.327100 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:25:53.327070 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd50fe3_bca7_4aa8_8ba6_75226b0adfdf.slice/crio-e3bd40d7b9aa0580b71cd0c1525e3b725cc20cf4f3e765b72ebde98c34edb69a WatchSource:0}: Error finding container e3bd40d7b9aa0580b71cd0c1525e3b725cc20cf4f3e765b72ebde98c34edb69a: Status 404 returned error can't find the container with id e3bd40d7b9aa0580b71cd0c1525e3b725cc20cf4f3e765b72ebde98c34edb69a Nov 03 20:25:53.327983 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:53.327933 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz"] Nov 03 20:25:53.330680 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:53.330648 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:53.330762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:53.330719 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:53.330762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:53.330750 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:53.341252 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:53.341225 4457 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" start-of-body= Nov 03 20:25:53.341360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:53.341286 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" podUID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.190:15021/healthz/ready\": dial tcp 10.133.0.190:15021: connect: connection refused" Nov 03 20:25:54.232363 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:54.232317 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" event={"ID":"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf","Type":"ContainerStarted","Data":"4bb6d99741865b9cbfbeacd8f5a623be4dd0b016385e261544fb08e92037bf5d"} Nov 03 20:25:54.232878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:54.232437 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" event={"ID":"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf","Type":"ContainerStarted","Data":"e3bd40d7b9aa0580b71cd0c1525e3b725cc20cf4f3e765b72ebde98c34edb69a"} Nov 03 20:25:54.262083 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:54.262020 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" podStartSLOduration=2.262000176 podStartE2EDuration="2.262000176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:54.258799211 +0000 UTC m=+1732.634395958" watchObservedRunningTime="2026-11-03 20:25:54.262000176 +0000 UTC m=+1732.637596926" Nov 03 20:25:54.346750 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:54.346716 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:54.347142 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:54.347117 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:54.347949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:54.347924 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:25:54.978953 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:54.978919 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:25:54.983663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:54.983629 4457 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 03 20:25:54.983899 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:54.983872 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" podUID="1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 03 20:25:55.209604 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.209562 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw"] Nov 03 20:25:55.273467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.273032 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw"] Nov 03 20:25:55.273467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.273179 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.290512 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.290477 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-77e83b-openshift-default-dockercfg-khxxv\"" Nov 03 20:25:55.308631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.308586 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-istio-token\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.308807 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.308733 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-socket\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.308807 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.308793 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-certs\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.308955 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.308828 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcbqf\" (UniqueName: \"kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-kube-api-access-gcbqf\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.309056 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.309035 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-credential-socket\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.309143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.309100 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-envoy\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.309289 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.309269 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-data\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.309454 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.309432 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32389e55-4f71-49b0-ba54-d53d241843b1-istiod-ca-cert\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.309528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.309512 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32389e55-4f71-49b0-ba54-d53d241843b1-istio-podinfo\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.414042 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.414004 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-certs\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.414239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.414069 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcbqf\" (UniqueName: \"kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-kube-api-access-gcbqf\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.414239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.414214 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-credential-socket\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.414364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.414241 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-envoy\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.414441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.414376 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-data\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.415644 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.415611 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32389e55-4f71-49b0-ba54-d53d241843b1-istiod-ca-cert\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.415789 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.415683 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32389e55-4f71-49b0-ba54-d53d241843b1-istio-podinfo\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.415789 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.415757 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-istio-token\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.415910 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.415894 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-socket\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.416456 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.416431 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-socket\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.416739 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.416715 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-certs\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.417011 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.416987 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-data\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.417861 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.417826 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-credential-socket\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.417947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.417900 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32389e55-4f71-49b0-ba54-d53d241843b1-istiod-ca-cert\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.421860 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.421829 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32389e55-4f71-49b0-ba54-d53d241843b1-istio-podinfo\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.422225 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.422204 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-envoy\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.430075 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.430051 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-istio-token\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.430474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.430455 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcbqf\" (UniqueName: \"kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-kube-api-access-gcbqf\") pod \"e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.704206 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.704162 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:55.994470 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.993980 4457 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 03 20:25:55.994470 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:55.994054 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" podUID="1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 03 20:25:56.047635 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:25:56.047605 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32389e55_4f71_49b0_ba54_d53d241843b1.slice/crio-0b49f5da8f0ea22fa6342e13cf5b0076557266e5611c8fe859c52233695605b1 WatchSource:0}: Error finding container 0b49f5da8f0ea22fa6342e13cf5b0076557266e5611c8fe859c52233695605b1: Status 404 returned error can't find the container with id 0b49f5da8f0ea22fa6342e13cf5b0076557266e5611c8fe859c52233695605b1 Nov 03 20:25:56.048232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:56.048206 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw"] Nov 03 20:25:56.051678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:56.051645 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:56.051792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:56.051738 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:56.051792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:56.051786 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:25:56.307427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:56.305226 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" event={"ID":"32389e55-4f71-49b0-ba54-d53d241843b1","Type":"ContainerStarted","Data":"318b199b14fa9e62ef7bd6af9bd6e35f916aba5d5806ab14da31e176a139813a"} Nov 03 20:25:56.307427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:56.305286 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" event={"ID":"32389e55-4f71-49b0-ba54-d53d241843b1","Type":"ContainerStarted","Data":"0b49f5da8f0ea22fa6342e13cf5b0076557266e5611c8fe859c52233695605b1"} Nov 03 20:25:56.396406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:56.395937 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" podStartSLOduration=1.395915955 podStartE2EDuration="1.395915955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:56.39291958 +0000 UTC m=+1734.768516330" watchObservedRunningTime="2026-11-03 20:25:56.395915955 +0000 UTC m=+1734.771512703" Nov 03 20:25:56.717467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:56.716840 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:25:56.967413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:56.963762 4457 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 03 20:25:56.967413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:56.963822 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" podUID="1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 03 20:25:57.216661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.216611 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4"] Nov 03 20:25:57.230450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.222880 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.237113 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.236972 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7ae0f1\"/\"test-model-7ae0f1-kserve-self-signed-certs\"" Nov 03 20:25:57.237277 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.237259 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7ae0f1\"/\"default-dockercfg-6f7b5\"" Nov 03 20:25:57.237978 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.237650 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4"] Nov 03 20:25:57.261413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.251981 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7ae0f1\"/\"kube-root-ca.crt\"" Nov 03 20:25:57.261413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.254757 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7ae0f1\"/\"openshift-service-ca.crt\"" Nov 03 20:25:57.468561 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.452987 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-home\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.468561 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.453043 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tls-certs\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.468561 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.453263 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-dshm\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.468561 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.453288 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25tbn\" (UniqueName: \"kubernetes.io/projected/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-kube-api-access-25tbn\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.492411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.482682 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-model-cache\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.492411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.482745 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tmp-dir\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.598416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.585607 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-home\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.614545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.612902 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tls-certs\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.614545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.613214 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-dshm\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.614545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.613248 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25tbn\" (UniqueName: \"kubernetes.io/projected/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-kube-api-access-25tbn\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.614545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.613550 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-model-cache\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.614545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.613589 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tmp-dir\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.614545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.614085 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tmp-dir\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.614545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.606934 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-home\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.632740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.632708 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-model-cache\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.687436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.681383 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-dshm\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.687436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.682049 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tls-certs\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.719002 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.709100 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25tbn\" (UniqueName: \"kubernetes.io/projected/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-kube-api-access-25tbn\") pod \"test-model-7ae0f1-kserve-84db68d48c-f8dr4\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.748413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.740717 4457 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 03 20:25:57.748413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.740941 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" podUID="32389e55-4f71-49b0-ba54-d53d241843b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 03 20:25:57.927411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.927226 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:57.988476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.983848 4457 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 03 20:25:57.988476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:57.983920 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" podUID="1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 03 20:25:58.178909 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:58.177600 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" Nov 03 20:25:58.707882 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:58.707791 4457 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 03 20:25:58.708435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:58.708401 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" podUID="32389e55-4f71-49b0-ba54-d53d241843b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 03 20:25:58.933483 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:58.933448 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4"] Nov 03 20:25:58.965556 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:58.965480 4457 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 03 20:25:58.965774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:58.965745 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" podUID="1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 03 20:25:59.134931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:59.134907 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" Nov 03 20:25:59.621965 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:59.621874 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" event={"ID":"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e","Type":"ContainerStarted","Data":"3e1d405ac8674758d132a1f74b4d5f9bfe309adce1f8227f82339b759127f348"} Nov 03 20:25:59.622273 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:59.622253 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" event={"ID":"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e","Type":"ContainerStarted","Data":"5d639ba72bcc7c517389e5d2b9e3e5a29863d56298059385d8c267a5960794ae"} Nov 03 20:25:59.687423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:59.685617 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" podStartSLOduration=2.685595161 podStartE2EDuration="2.685595161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:25:59.682419622 +0000 UTC m=+1738.058016369" watchObservedRunningTime="2026-11-03 20:25:59.685595161 +0000 UTC m=+1738.061191909" Nov 03 20:25:59.726459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:59.726112 4457 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 03 20:25:59.726459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:59.726170 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" podUID="32389e55-4f71-49b0-ba54-d53d241843b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 03 20:25:59.946167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:59.946070 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:25:59.961413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:59.960300 4457 patch_prober.go:29] interesting pod/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 03 20:25:59.961413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:25:59.960404 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" podUID="1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 03 20:26:00.705233 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:00.705191 4457 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 03 20:26:00.705463 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:00.705260 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" podUID="32389e55-4f71-49b0-ba54-d53d241843b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 03 20:26:01.191417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:01.190316 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:26:01.191900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:01.191873 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:26:01.195714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:01.195690 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:26:01.704804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:01.704765 4457 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 03 20:26:01.704989 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:01.704820 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" podUID="32389e55-4f71-49b0-ba54-d53d241843b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 03 20:26:02.705656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:02.705530 4457 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 03 20:26:02.705656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:02.705594 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" podUID="32389e55-4f71-49b0-ba54-d53d241843b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 03 20:26:03.704980 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:03.704943 4457 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 03 20:26:03.705180 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:03.705000 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" podUID="32389e55-4f71-49b0-ba54-d53d241843b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 03 20:26:04.711639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:04.711606 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:26:04.712207 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:04.712173 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:26:04.712906 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:04.712885 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:26:06.149313 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:26:06.149259 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:26:07.928263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:07.928230 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:26:07.941335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:07.941302 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:26:08.812061 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:08.812030 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:26:17.928228 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:17.928134 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj"] Nov 03 20:26:17.928682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:17.928368 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" containerID="cri-o://639bc6eae804a3afbb34e0e83beed35a0196da389ac24057ab8f12b321164f75" gracePeriod=30 Nov 03 20:26:17.935120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:17.935089 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq"] Nov 03 20:26:17.935364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:17.935311 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerName="istio-proxy" containerID="cri-o://002132b6698812c012b45b3990d7f780ee7659ea5e86d3b7bcbe5c3d7fab9987" gracePeriod=30 Nov 03 20:26:23.312342 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.312015 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:26:23.331238 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.331208 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:26:23.480955 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.480913 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-podinfo\") pod \"10eb980a-3ddd-4425-9636-7b44724c76ed\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " Nov 03 20:26:23.481163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.480976 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-socket\") pod \"086323f3-b415-4716-94d4-ea06f7ada7ee\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " Nov 03 20:26:23.481163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481090 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-kube-api-access-cj57r\" (UniqueName: \"kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-kube-api-access-cj57r\") pod \"086323f3-b415-4716-94d4-ea06f7ada7ee\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " Nov 03 20:26:23.481163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481156 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-credential-socket\") pod \"086323f3-b415-4716-94d4-ea06f7ada7ee\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " Nov 03 20:26:23.481318 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481187 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10eb980a-3ddd-4425-9636-7b44724c76ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10eb980a-3ddd-4425-9636-7b44724c76ed-istiod-ca-cert\") pod \"10eb980a-3ddd-4425-9636-7b44724c76ed\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " Nov 03 20:26:23.481318 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481223 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/086323f3-b415-4716-94d4-ea06f7ada7ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086323f3-b415-4716-94d4-ea06f7ada7ee-istiod-ca-cert\") pod \"086323f3-b415-4716-94d4-ea06f7ada7ee\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " Nov 03 20:26:23.481318 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481298 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-socket" pod "086323f3-b415-4716-94d4-ea06f7ada7ee" (UID: "086323f3-b415-4716-94d4-ea06f7ada7ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.481536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481309 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-podinfo\") pod \"086323f3-b415-4716-94d4-ea06f7ada7ee\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " Nov 03 20:26:23.481591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481529 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-token\" (UniqueName: \"kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-token\") pod \"086323f3-b415-4716-94d4-ea06f7ada7ee\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " Nov 03 20:26:23.481645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481631 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-data\") pod \"086323f3-b415-4716-94d4-ea06f7ada7ee\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " Nov 03 20:26:23.481699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481661 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-token\" (UniqueName: \"kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-token\") pod \"10eb980a-3ddd-4425-9636-7b44724c76ed\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " Nov 03 20:26:23.481756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481700 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-data\") pod \"10eb980a-3ddd-4425-9636-7b44724c76ed\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " Nov 03 20:26:23.481756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481740 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-envoy\") pod \"10eb980a-3ddd-4425-9636-7b44724c76ed\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " Nov 03 20:26:23.481864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481799 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-certs\") pod \"086323f3-b415-4716-94d4-ea06f7ada7ee\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " Nov 03 20:26:23.481864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481844 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-certs\") pod \"10eb980a-3ddd-4425-9636-7b44724c76ed\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " Nov 03 20:26:23.481963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481887 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-socket\") pod \"10eb980a-3ddd-4425-9636-7b44724c76ed\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " Nov 03 20:26:23.481963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481926 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-credential-socket\") pod \"10eb980a-3ddd-4425-9636-7b44724c76ed\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " Nov 03 20:26:23.481963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481953 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-kube-api-access-hbzgw\" (UniqueName: \"kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-kube-api-access-hbzgw\") pod \"10eb980a-3ddd-4425-9636-7b44724c76ed\" (UID: \"10eb980a-3ddd-4425-9636-7b44724c76ed\") " Nov 03 20:26:23.482115 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.482079 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-envoy\") pod \"086323f3-b415-4716-94d4-ea06f7ada7ee\" (UID: \"086323f3-b415-4716-94d4-ea06f7ada7ee\") " Nov 03 20:26:23.483297 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.483019 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.483481 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.481628 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-credential-socket" pod "086323f3-b415-4716-94d4-ea06f7ada7ee" (UID: "086323f3-b415-4716-94d4-ea06f7ada7ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.483557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.482105 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10eb980a-3ddd-4425-9636-7b44724c76ed-istiod-ca-cert" pod "10eb980a-3ddd-4425-9636-7b44724c76ed" (UID: "10eb980a-3ddd-4425-9636-7b44724c76ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:23.483557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.482851 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-socket" pod "10eb980a-3ddd-4425-9636-7b44724c76ed" (UID: "10eb980a-3ddd-4425-9636-7b44724c76ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.483557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.483141 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086323f3-b415-4716-94d4-ea06f7ada7ee-istiod-ca-cert" pod "086323f3-b415-4716-94d4-ea06f7ada7ee" (UID: "086323f3-b415-4716-94d4-ea06f7ada7ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:26:23.483557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.483447 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-certs" pod "086323f3-b415-4716-94d4-ea06f7ada7ee" (UID: "086323f3-b415-4716-94d4-ea06f7ada7ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.483855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.483830 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-certs" pod "10eb980a-3ddd-4425-9636-7b44724c76ed" (UID: "10eb980a-3ddd-4425-9636-7b44724c76ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.484334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.483958 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-podinfo" pod "086323f3-b415-4716-94d4-ea06f7ada7ee" (UID: "086323f3-b415-4716-94d4-ea06f7ada7ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:23.484334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.484131 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-podinfo" pod "10eb980a-3ddd-4425-9636-7b44724c76ed" (UID: "10eb980a-3ddd-4425-9636-7b44724c76ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:26:23.484334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.484137 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-data" pod "086323f3-b415-4716-94d4-ea06f7ada7ee" (UID: "086323f3-b415-4716-94d4-ea06f7ada7ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.484334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.484197 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-credential-socket" pod "10eb980a-3ddd-4425-9636-7b44724c76ed" (UID: "10eb980a-3ddd-4425-9636-7b44724c76ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.484334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.484267 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-kube-api-access-cj57r" pod "086323f3-b415-4716-94d4-ea06f7ada7ee" (UID: "086323f3-b415-4716-94d4-ea06f7ada7ee"). InnerVolumeSpecName "kube-api-access-cj57r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:23.484334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.484314 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-data" pod "10eb980a-3ddd-4425-9636-7b44724c76ed" (UID: "10eb980a-3ddd-4425-9636-7b44724c76ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.486119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.486092 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-token" pod "086323f3-b415-4716-94d4-ea06f7ada7ee" (UID: "086323f3-b415-4716-94d4-ea06f7ada7ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:23.486421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.486309 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-envoy" pod "10eb980a-3ddd-4425-9636-7b44724c76ed" (UID: "10eb980a-3ddd-4425-9636-7b44724c76ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.486421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.486374 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-envoy" pod "086323f3-b415-4716-94d4-ea06f7ada7ee" (UID: "086323f3-b415-4716-94d4-ea06f7ada7ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:26:23.486945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.486920 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-kube-api-access-hbzgw" pod "10eb980a-3ddd-4425-9636-7b44724c76ed" (UID: "10eb980a-3ddd-4425-9636-7b44724c76ed"). InnerVolumeSpecName "kube-api-access-hbzgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:23.487848 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.487821 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-token" pod "10eb980a-3ddd-4425-9636-7b44724c76ed" (UID: "10eb980a-3ddd-4425-9636-7b44724c76ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:26:23.584615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584577 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj57r\" (UniqueName: \"kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-kube-api-access-cj57r\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584619 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584637 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10eb980a-3ddd-4425-9636-7b44724c76ed-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584652 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086323f3-b415-4716-94d4-ea06f7ada7ee-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584667 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584681 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584694 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584706 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584721 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584735 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584748 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584761 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584776 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584789 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eb980a-3ddd-4425-9636-7b44724c76ed-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584803 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbzgw\" (UniqueName: \"kubernetes.io/projected/10eb980a-3ddd-4425-9636-7b44724c76ed-kube-api-access-hbzgw\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584817 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086323f3-b415-4716-94d4-ea06f7ada7ee-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.584891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.584830 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10eb980a-3ddd-4425-9636-7b44724c76ed-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:26:23.609863 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.609819 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56"] Nov 03 20:26:23.610901 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.610880 4457 cpu_manager.go:414] "containerMap: removing container" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" Nov 03 20:26:23.610901 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.610902 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" Nov 03 20:26:23.611059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.610918 4457 cpu_manager.go:414] "containerMap: removing container" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerName="istio-proxy" Nov 03 20:26:23.611059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.610924 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerName="istio-proxy" Nov 03 20:26:23.611178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.611081 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerName="istio-proxy" Nov 03 20:26:23.611178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.611091 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerName="istio-proxy" Nov 03 20:26:23.616964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.616940 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.626347 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.625376 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b0adda\"/\"test-model-b0adda-kserve-self-signed-certs\"" Nov 03 20:26:23.626347 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.625503 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56"] Nov 03 20:26:23.626347 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.625565 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-b0adda\"/\"default-dockercfg-dlqb4\"" Nov 03 20:26:23.688211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.687604 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b0adda\"/\"openshift-service-ca.crt\"" Nov 03 20:26:23.688211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.687712 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-b0adda\"/\"kube-root-ca.crt\"" Nov 03 20:26:23.843709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.836198 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tmp-dir\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.843709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.836324 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-dshm\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.843709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.836378 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-home\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.843709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.836450 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4jbk\" (UniqueName: \"kubernetes.io/projected/2699c9a3-baf0-40f0-b1be-892b29a62ad4-kube-api-access-r4jbk\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.843709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.836531 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tls-certs\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.843709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.836767 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-model-cache\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.945417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.942890 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-model-cache\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.945417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.942963 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tmp-dir\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.945417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.943062 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-dshm\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.945417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.943109 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-home\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.945417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.943162 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4jbk\" (UniqueName: \"kubernetes.io/projected/2699c9a3-baf0-40f0-b1be-892b29a62ad4-kube-api-access-r4jbk\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.945417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.943240 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tls-certs\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.968865 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.968609 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-home\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.969371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.969349 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-model-cache\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.972339 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.972287 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tmp-dir\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:23.979433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:23.973245 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tls-certs\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:24.028643 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.027760 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-dshm\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:24.043430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.042265 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4jbk\" (UniqueName: \"kubernetes.io/projected/2699c9a3-baf0-40f0-b1be-892b29a62ad4-kube-api-access-r4jbk\") pod \"test-model-b0adda-kserve-65d7cc555-ddt56\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:24.065537 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.065109 4457 generic.go:356] "Generic (PLEG): container finished" podID="086323f3-b415-4716-94d4-ea06f7ada7ee" containerID="639bc6eae804a3afbb34e0e83beed35a0196da389ac24057ab8f12b321164f75" exitCode=0 Nov 03 20:26:24.065742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.065615 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" event={"ID":"086323f3-b415-4716-94d4-ea06f7ada7ee","Type":"ContainerDied","Data":"639bc6eae804a3afbb34e0e83beed35a0196da389ac24057ab8f12b321164f75"} Nov 03 20:26:24.065742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.065675 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" event={"ID":"086323f3-b415-4716-94d4-ea06f7ada7ee","Type":"ContainerDied","Data":"135af36d2106f628e70d8cd039967cca3036131ac408a7e8f787feb589a64380"} Nov 03 20:26:24.065742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.065719 4457 scope.go:122] "RemoveContainer" containerID="639bc6eae804a3afbb34e0e83beed35a0196da389ac24057ab8f12b321164f75" Nov 03 20:26:24.065945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.065928 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj" Nov 03 20:26:24.087354 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.086281 4457 generic.go:356] "Generic (PLEG): container finished" podID="10eb980a-3ddd-4425-9636-7b44724c76ed" containerID="002132b6698812c012b45b3990d7f780ee7659ea5e86d3b7bcbe5c3d7fab9987" exitCode=0 Nov 03 20:26:24.087354 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.086355 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" event={"ID":"10eb980a-3ddd-4425-9636-7b44724c76ed","Type":"ContainerDied","Data":"002132b6698812c012b45b3990d7f780ee7659ea5e86d3b7bcbe5c3d7fab9987"} Nov 03 20:26:24.087354 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.086416 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" event={"ID":"10eb980a-3ddd-4425-9636-7b44724c76ed","Type":"ContainerDied","Data":"3e98af5b167a9904f8fc88339d2cabc3dabcd525473b160195941da49901ac0a"} Nov 03 20:26:24.087354 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.086505 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq" Nov 03 20:26:24.138661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.138466 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj"] Nov 03 20:26:24.152650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.145470 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-84kwj"] Nov 03 20:26:24.174315 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.173659 4457 scope.go:122] "RemoveContainer" containerID="639bc6eae804a3afbb34e0e83beed35a0196da389ac24057ab8f12b321164f75" Nov 03 20:26:24.175117 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:26:24.174666 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639bc6eae804a3afbb34e0e83beed35a0196da389ac24057ab8f12b321164f75\": container with ID starting with 639bc6eae804a3afbb34e0e83beed35a0196da389ac24057ab8f12b321164f75 not found: ID does not exist" containerID="639bc6eae804a3afbb34e0e83beed35a0196da389ac24057ab8f12b321164f75" Nov 03 20:26:24.175117 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.174703 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639bc6eae804a3afbb34e0e83beed35a0196da389ac24057ab8f12b321164f75"} err="failed to get container status \"639bc6eae804a3afbb34e0e83beed35a0196da389ac24057ab8f12b321164f75\": rpc error: code = NotFound desc = could not find container \"639bc6eae804a3afbb34e0e83beed35a0196da389ac24057ab8f12b321164f75\": container with ID starting with 639bc6eae804a3afbb34e0e83beed35a0196da389ac24057ab8f12b321164f75 not found: ID does not exist" Nov 03 20:26:24.175117 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.174730 4457 scope.go:122] "RemoveContainer" containerID="002132b6698812c012b45b3990d7f780ee7659ea5e86d3b7bcbe5c3d7fab9987" Nov 03 20:26:24.189915 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.187741 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq"] Nov 03 20:26:24.199550 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.199203 4457 scope.go:122] "RemoveContainer" containerID="002132b6698812c012b45b3990d7f780ee7659ea5e86d3b7bcbe5c3d7fab9987" Nov 03 20:26:24.199627 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:26:24.199608 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002132b6698812c012b45b3990d7f780ee7659ea5e86d3b7bcbe5c3d7fab9987\": container with ID starting with 002132b6698812c012b45b3990d7f780ee7659ea5e86d3b7bcbe5c3d7fab9987 not found: ID does not exist" containerID="002132b6698812c012b45b3990d7f780ee7659ea5e86d3b7bcbe5c3d7fab9987" Nov 03 20:26:24.199703 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.199640 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002132b6698812c012b45b3990d7f780ee7659ea5e86d3b7bcbe5c3d7fab9987"} err="failed to get container status \"002132b6698812c012b45b3990d7f780ee7659ea5e86d3b7bcbe5c3d7fab9987\": rpc error: code = NotFound desc = could not find container \"002132b6698812c012b45b3990d7f780ee7659ea5e86d3b7bcbe5c3d7fab9987\": container with ID starting with 002132b6698812c012b45b3990d7f780ee7659ea5e86d3b7bcbe5c3d7fab9987 not found: ID does not exist" Nov 03 20:26:24.201376 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.201257 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-2d6rq"] Nov 03 20:26:24.244726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.244689 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:24.553559 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:26:24.553526 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2699c9a3_baf0_40f0_b1be_892b29a62ad4.slice/crio-02a7bf7cf3332a3e2d7cacf1e9d91d4bf084aac59f6d5c98f777532411c6d531 WatchSource:0}: Error finding container 02a7bf7cf3332a3e2d7cacf1e9d91d4bf084aac59f6d5c98f777532411c6d531: Status 404 returned error can't find the container with id 02a7bf7cf3332a3e2d7cacf1e9d91d4bf084aac59f6d5c98f777532411c6d531 Nov 03 20:26:24.554485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:24.554456 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56"] Nov 03 20:26:25.116464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:25.116419 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" event={"ID":"2699c9a3-baf0-40f0-b1be-892b29a62ad4","Type":"ContainerStarted","Data":"8f849a8f2bc84270ebd0632220f6673d758c31db2320ffbbf01ccc9807cf0bf9"} Nov 03 20:26:25.116682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:25.116477 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" event={"ID":"2699c9a3-baf0-40f0-b1be-892b29a62ad4","Type":"ContainerStarted","Data":"02a7bf7cf3332a3e2d7cacf1e9d91d4bf084aac59f6d5c98f777532411c6d531"} Nov 03 20:26:25.140438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:25.140326 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" podStartSLOduration=2.1403066 podStartE2EDuration="2.1403066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:25.135918744 +0000 UTC m=+1763.511515513" watchObservedRunningTime="2026-11-03 20:26:25.1403066 +0000 UTC m=+1763.515903347" Nov 03 20:26:25.245455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:25.245415 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:25.763821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:25.763780 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086323f3-b415-4716-94d4-ea06f7ada7ee" path="/var/lib/kubelet/pods/086323f3-b415-4716-94d4-ea06f7ada7ee/volumes" Nov 03 20:26:25.764504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:25.764484 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10eb980a-3ddd-4425-9636-7b44724c76ed" path="/var/lib/kubelet/pods/10eb980a-3ddd-4425-9636-7b44724c76ed/volumes" Nov 03 20:26:29.537977 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:29.537938 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk"] Nov 03 20:26:29.548401 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:29.548282 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f3ae23-m4wzk"] Nov 03 20:26:29.765509 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:29.765455 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="592b66d4-4ef3-4941-b576-657982ac082e" path="/var/lib/kubelet/pods/592b66d4-4ef3-4941-b576-657982ac082e/volumes" Nov 03 20:26:34.246077 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:34.246038 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:34.261191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:34.261162 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:34.284021 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:34.283988 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:26:35.807435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:35.804553 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww"] Nov 03 20:26:35.845344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:35.845308 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww"] Nov 03 20:26:35.845576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:35.845449 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:35.854230 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:35.854201 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a744eb-openshift-default-dockercfg-n26cv\"" Nov 03 20:26:35.984419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:35.967364 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-certs\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.015707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.015668 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-credential-socket\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.016111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.016086 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d097968-5008-4658-a534-fe81ba625490-istio-podinfo\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.016421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.016363 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ff7d\" (UniqueName: \"kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-kube-api-access-8ff7d\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.016903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.016881 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-data\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.017132 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.017113 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d097968-5008-4658-a534-fe81ba625490-istiod-ca-cert\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.017406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.017323 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-socket\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.017583 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.017566 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-istio-token\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.017881 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.017859 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-envoy\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.210508 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.210411 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ff7d\" (UniqueName: \"kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-kube-api-access-8ff7d\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.268167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.268052 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-data\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.269793 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.269764 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d097968-5008-4658-a534-fe81ba625490-istiod-ca-cert\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.286609 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.285524 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-data\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.288776 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.288713 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ff7d\" (UniqueName: \"kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-kube-api-access-8ff7d\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.291334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.291291 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d097968-5008-4658-a534-fe81ba625490-istiod-ca-cert\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.291799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.291625 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-socket\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.291972 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.291957 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-istio-token\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.292339 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.292323 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-envoy\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.293257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.292729 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-socket\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.297148 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.297023 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-certs\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.298166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.297722 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-certs\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.298343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.298325 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-credential-socket\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.298900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.298879 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d097968-5008-4658-a534-fe81ba625490-istio-podinfo\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.299326 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.298771 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-credential-socket\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.309631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.309601 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-istio-token\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.313997 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.313974 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-envoy\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.325646 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:26:36.322591 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:26:36.328892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.328858 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d097968-5008-4658-a534-fe81ba625490-istio-podinfo\") pod \"e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.477141 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.477041 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:36.842060 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.841997 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww"] Nov 03 20:26:36.845158 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:26:36.845125 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d097968_5008_4658_a534_fe81ba625490.slice/crio-76ab52b7ff26c06813b6ba5c20234b0573dd5e63764223ef331ef30e582f14e6 WatchSource:0}: Error finding container 76ab52b7ff26c06813b6ba5c20234b0573dd5e63764223ef331ef30e582f14e6: Status 404 returned error can't find the container with id 76ab52b7ff26c06813b6ba5c20234b0573dd5e63764223ef331ef30e582f14e6 Nov 03 20:26:36.848223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.848181 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:26:36.848291 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.848252 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:26:36.848291 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:36.848285 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:26:37.416409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:37.416358 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" event={"ID":"7d097968-5008-4658-a534-fe81ba625490","Type":"ContainerStarted","Data":"4496f6ade23dd61b318645f2d4d985651b69df45c44dbf2265a9dec503169dfc"} Nov 03 20:26:37.416614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:37.416469 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" event={"ID":"7d097968-5008-4658-a534-fe81ba625490","Type":"ContainerStarted","Data":"76ab52b7ff26c06813b6ba5c20234b0573dd5e63764223ef331ef30e582f14e6"} Nov 03 20:26:37.444275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:37.444189 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" podStartSLOduration=2.444168435 podStartE2EDuration="2.444168435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:37.441435257 +0000 UTC m=+1775.817032005" watchObservedRunningTime="2026-11-03 20:26:37.444168435 +0000 UTC m=+1775.819765183" Nov 03 20:26:37.477217 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:37.477166 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:37.478894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:37.478863 4457 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:26:37.479024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:37.478932 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 20:26:38.050734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:38.050698 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl"] Nov 03 20:26:38.058474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:38.057937 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f19886e6-5mdtl"] Nov 03 20:26:38.673414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:38.673001 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:39.516430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:39.516395 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:39.517460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:39.517439 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:26:39.768172 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:39.763662 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efe61317-8ea3-451b-ad9a-24678f00034f" path="/var/lib/kubelet/pods/efe61317-8ea3-451b-ad9a-24678f00034f/volumes" Nov 03 20:26:41.387510 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:41.387473 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj"] Nov 03 20:26:41.393527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:41.393495 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" Nov 03 20:26:41.399062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:41.399013 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a744eb-dockercfg-rllpz\"" Nov 03 20:26:41.399223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:41.399152 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a744eb\"" Nov 03 20:26:41.401821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:41.401790 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj"] Nov 03 20:26:41.450970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:41.450937 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcs8g\" (UniqueName: \"kubernetes.io/projected/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-kube-api-access-mcs8g\") pod \"payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj\" (UID: \"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" Nov 03 20:26:41.451158 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:41.451132 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj\" (UID: \"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" Nov 03 20:26:41.609407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:41.601840 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcs8g\" (UniqueName: \"kubernetes.io/projected/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-kube-api-access-mcs8g\") pod \"payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj\" (UID: \"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" Nov 03 20:26:41.609407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:41.602209 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj\" (UID: \"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" Nov 03 20:26:41.609407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:41.603743 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj\" (UID: \"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" Nov 03 20:26:41.741995 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:41.740695 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcs8g\" (UniqueName: \"kubernetes.io/projected/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-kube-api-access-mcs8g\") pod \"payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj\" (UID: \"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" Nov 03 20:26:42.020359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:42.020274 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" Nov 03 20:26:42.338162 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:42.335360 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj"] Nov 03 20:26:42.949570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:42.949529 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" event={"ID":"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f","Type":"ContainerStarted","Data":"5b8310b3086cae90cbcb030e36a6cbee22bb8291b5b8a824d397cfe07f2f1ce4"} Nov 03 20:26:42.950008 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:42.949587 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" event={"ID":"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f","Type":"ContainerStarted","Data":"0f91ee99c8cab2d2e458b65bcbb8fc0ee3ed91cd69bc183de095f8276ba0f1c9"} Nov 03 20:26:42.950008 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:42.949800 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" Nov 03 20:26:42.970138 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:42.970071 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" podStartSLOduration=1.970054145 podStartE2EDuration="1.970054145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:42.967496373 +0000 UTC m=+1781.343093122" watchObservedRunningTime="2026-11-03 20:26:42.970054145 +0000 UTC m=+1781.345650892" Nov 03 20:26:43.368852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:43.368607 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl"] Nov 03 20:26:43.375491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:43.375154 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" Nov 03 20:26:43.382009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:43.381979 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-45e569\"" Nov 03 20:26:43.383296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:43.383266 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl"] Nov 03 20:26:43.401715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:43.401683 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-45e569-dockercfg-52dfw\"" Nov 03 20:26:43.437313 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:43.437274 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cbe749-2e98-4766-b527-8aa40a28e2b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl\" (UID: \"37cbe749-2e98-4766-b527-8aa40a28e2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" Nov 03 20:26:43.437624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:43.437520 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbkx\" (UniqueName: \"kubernetes.io/projected/37cbe749-2e98-4766-b527-8aa40a28e2b2-kube-api-access-jdbkx\") pod \"payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl\" (UID: \"37cbe749-2e98-4766-b527-8aa40a28e2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" Nov 03 20:26:43.539376 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:43.539333 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cbe749-2e98-4766-b527-8aa40a28e2b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl\" (UID: \"37cbe749-2e98-4766-b527-8aa40a28e2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" Nov 03 20:26:43.539627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:43.539607 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbkx\" (UniqueName: \"kubernetes.io/projected/37cbe749-2e98-4766-b527-8aa40a28e2b2-kube-api-access-jdbkx\") pod \"payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl\" (UID: \"37cbe749-2e98-4766-b527-8aa40a28e2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" Nov 03 20:26:43.540119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:43.540095 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cbe749-2e98-4766-b527-8aa40a28e2b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl\" (UID: \"37cbe749-2e98-4766-b527-8aa40a28e2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" Nov 03 20:26:43.553738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:43.553707 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbkx\" (UniqueName: \"kubernetes.io/projected/37cbe749-2e98-4766-b527-8aa40a28e2b2-kube-api-access-jdbkx\") pod \"payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl\" (UID: \"37cbe749-2e98-4766-b527-8aa40a28e2b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" Nov 03 20:26:43.711524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:43.709021 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" Nov 03 20:26:44.235301 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:26:44.235270 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37cbe749_2e98_4766_b527_8aa40a28e2b2.slice/crio-3aafdcd282c793e7205f8847a70cf21cb58869a759cb216b11bf72a048230e14 WatchSource:0}: Error finding container 3aafdcd282c793e7205f8847a70cf21cb58869a759cb216b11bf72a048230e14: Status 404 returned error can't find the container with id 3aafdcd282c793e7205f8847a70cf21cb58869a759cb216b11bf72a048230e14 Nov 03 20:26:44.235753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:44.235730 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl"] Nov 03 20:26:44.985627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:44.985584 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" event={"ID":"37cbe749-2e98-4766-b527-8aa40a28e2b2","Type":"ContainerStarted","Data":"b31f021a3ae05450b2ab3fc5bd28d1643ed1e0ea199c21739bc5174498435547"} Nov 03 20:26:44.985840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:44.985664 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" event={"ID":"37cbe749-2e98-4766-b527-8aa40a28e2b2","Type":"ContainerStarted","Data":"3aafdcd282c793e7205f8847a70cf21cb58869a759cb216b11bf72a048230e14"} Nov 03 20:26:44.985840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:44.985714 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" Nov 03 20:26:45.004672 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:45.004610 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" podStartSLOduration=2.004593873 podStartE2EDuration="2.004593873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:26:45.004112843 +0000 UTC m=+1783.379709591" watchObservedRunningTime="2026-11-03 20:26:45.004593873 +0000 UTC m=+1783.380190620" Nov 03 20:26:53.970222 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:53.970190 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" Nov 03 20:26:56.001220 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:26:56.001189 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" Nov 03 20:27:02.636180 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:02.636146 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:27:02.639492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:02.639463 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:27:02.654475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:02.654453 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:27:02.673780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:02.673752 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:27:06.234246 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:27:06.232617 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:27:06.985184 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:06.985142 4457 scope.go:122] "RemoveContainer" containerID="72025990159b59d0d715c66f59d6a42348872aef070cdea19ca0adff51c96656" Nov 03 20:27:07.008312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:07.008284 4457 scope.go:122] "RemoveContainer" containerID="1ad13c4e09308f4c437f09b11f20ed148b57f7e19c3c7491de04ebbfc8b326e0" Nov 03 20:27:07.022622 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:07.022600 4457 scope.go:122] "RemoveContainer" containerID="b40e7ac2c90415020d0daf3d9d3969ac94b25fe60e538392e3ed21499cd2a55d" Nov 03 20:27:08.950454 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:08.932756 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" podUID="a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:08.972411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:08.955458 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:08.976409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:08.974457 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" podUID="5d2da67e-e41e-441b-9a9d-1272bc5726a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.180:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.013425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.005007 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.143435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.139462 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.234492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.223695 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Nov 03 20:27:09.544489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.531248 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.544489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.531747 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.578357 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.578314 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.598415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.597529 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.720528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.720480 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.789411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.780161 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.823516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.823472 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" podUID="a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.866557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.866517 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p7lrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:27:09.866846 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.866812 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" podUID="93f7f2c5-96a8-4ccc-ad9b-3cc648f85802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:09.975805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.975751 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" podUID="5d2da67e-e41e-441b-9a9d-1272bc5726a4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.180:8000/ready\": context deadline exceeded" Nov 03 20:27:09.984069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:09.984036 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:27:10.055002 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.054955 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:10.096463 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.071016 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:27:10.259411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.141507 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:10.309612 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.308517 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" podUID="2699c9a3-baf0-40f0-b1be-892b29a62ad4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.194:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:10.399273 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.399173 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 03 20:27:10.534696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.533184 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.89:8000/ready\": context deadline exceeded" Nov 03 20:27:10.534696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.533801 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:27:10.552335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.552292 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:10.553947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.553921 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:27:10.576808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.576777 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:27:10.580525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.580484 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:10.580792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.580773 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:27:10.604349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.604303 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:27:10.605155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.605124 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:27:10.700506 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.687327 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:27:10.700975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.700946 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:27:10.726585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.726541 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Nov 03 20:27:10.728994 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.728967 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:27:10.756362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.756333 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:27:10.757736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.757714 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:27:10.780036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.779996 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 03 20:27:10.844456 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.844249 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:27:10.874681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:10.874182 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:27:32.772325 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:32.772285 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb"] Nov 03 20:27:32.789690 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:32.789659 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb"] Nov 03 20:27:32.790030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:32.789928 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:32.796343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:32.795092 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c2c601\"/\"default-dockercfg-6x2ls\"" Nov 03 20:27:32.805450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:32.803489 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c2c601\"/\"test-model-c2c601-kserve-self-signed-certs\"" Nov 03 20:27:32.805450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:32.804568 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c2c601\"/\"openshift-service-ca.crt\"" Nov 03 20:27:32.805450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:32.804595 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c2c601\"/\"kube-root-ca.crt\"" Nov 03 20:27:32.984952 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:32.971765 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-dshm\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:32.985491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:32.985453 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-tmp-dir\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:32.985799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:32.985777 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-home\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:32.985956 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:32.985940 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a384ec-5e7f-4f72-9381-92b564160ef6-tls-certs\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.017410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:32.986215 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-model-cache\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.017410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.001829 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzdlk\" (UniqueName: \"kubernetes.io/projected/75a384ec-5e7f-4f72-9381-92b564160ef6-kube-api-access-kzdlk\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.105960 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.105919 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a384ec-5e7f-4f72-9381-92b564160ef6-tls-certs\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.106199 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.106100 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-model-cache\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.106199 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.106177 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzdlk\" (UniqueName: \"kubernetes.io/projected/75a384ec-5e7f-4f72-9381-92b564160ef6-kube-api-access-kzdlk\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.106377 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.106353 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-dshm\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.106921 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.106892 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-model-cache\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.107849 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.107824 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-tmp-dir\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.107966 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.107947 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-home\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.108411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.108369 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-home\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.108758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.108736 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-tmp-dir\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.111758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.111735 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-dshm\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.130181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.130142 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzdlk\" (UniqueName: \"kubernetes.io/projected/75a384ec-5e7f-4f72-9381-92b564160ef6-kube-api-access-kzdlk\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.145738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.145699 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a384ec-5e7f-4f72-9381-92b564160ef6-tls-certs\") pod \"test-model-c2c601-kserve-84f4db5bcc-xlzsb\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.420011 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.419931 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:33.579826 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.579136 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf"] Nov 03 20:27:33.580437 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.580377 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" podUID="49cbfb85-a563-4265-828a-40ead144734b" containerName="istio-proxy" containerID="cri-o://f372bd35881b53c4a32db85199fc7910b04d16c8ae172f84953836040e55d9d8" gracePeriod=30 Nov 03 20:27:33.747211 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:27:33.747167 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75a384ec_5e7f_4f72_9381_92b564160ef6.slice/crio-fb7adf5e8c308e1ab82ae38fc23eb54e59ae726e326eac2fec8c05ab65ccf70b WatchSource:0}: Error finding container fb7adf5e8c308e1ab82ae38fc23eb54e59ae726e326eac2fec8c05ab65ccf70b: Status 404 returned error can't find the container with id fb7adf5e8c308e1ab82ae38fc23eb54e59ae726e326eac2fec8c05ab65ccf70b Nov 03 20:27:33.749263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:33.749195 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb"] Nov 03 20:27:34.311410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:34.311271 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" event={"ID":"75a384ec-5e7f-4f72-9381-92b564160ef6","Type":"ContainerStarted","Data":"6333a4b6dfbff259d2434d3ea39c743cd094ba533e4a1befc379a4fc2888ac4e"} Nov 03 20:27:34.311410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:34.311329 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" event={"ID":"75a384ec-5e7f-4f72-9381-92b564160ef6","Type":"ContainerStarted","Data":"fb7adf5e8c308e1ab82ae38fc23eb54e59ae726e326eac2fec8c05ab65ccf70b"} Nov 03 20:27:34.374408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:34.368204 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" podStartSLOduration=2.368186874 podStartE2EDuration="2.368186874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:34.363124451 +0000 UTC m=+1832.738721200" watchObservedRunningTime="2026-11-03 20:27:34.368186874 +0000 UTC m=+1832.743783626" Nov 03 20:27:34.427409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:34.426493 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:36.157171 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:27:36.157107 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:27:38.547256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.547218 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22"] Nov 03 20:27:38.553362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.553336 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.561051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.560110 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7ae0f1-openshift-default-dockercfg-xwqqn\"" Nov 03 20:27:38.569007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.568979 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22"] Nov 03 20:27:38.631853 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.631816 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-envoy\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.632004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.631905 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.632195 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.632160 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-certs\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.632333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.632257 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-data\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.632333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.632297 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istiod-ca-cert\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.632535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.632349 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-podinfo\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.632535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.632398 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-token\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.632743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.632564 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62q5c\" (UniqueName: \"kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-kube-api-access-62q5c\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.632743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.632701 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-credential-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.734802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.734758 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-credential-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.735008 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.734854 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-envoy\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.735008 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.734927 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.735196 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.735146 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-certs\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.735247 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.735233 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-credential-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.735319 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.735303 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-data\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.735357 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.735348 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istiod-ca-cert\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.735428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.735412 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-podinfo\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.735482 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.735452 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-token\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.735570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.735542 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62q5c\" (UniqueName: \"kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-kube-api-access-62q5c\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.735957 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.735932 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-certs\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.736133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.736107 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-data\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.736222 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.736161 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-socket\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.736291 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.736258 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istiod-ca-cert\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.737749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.737722 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-envoy\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.738322 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.738299 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-podinfo\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.765718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.765675 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-token\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.766325 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.766301 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62q5c\" (UniqueName: \"kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-kube-api-access-62q5c\") pod \"e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.884407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.878709 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:38.957671 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:38.957644 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:27:39.043944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.043903 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-credential-socket\") pod \"49cbfb85-a563-4265-828a-40ead144734b\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " Nov 03 20:27:39.044142 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.043980 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-envoy\") pod \"49cbfb85-a563-4265-828a-40ead144734b\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " Nov 03 20:27:39.044142 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.044071 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-kube-api-access-7qk99\" (UniqueName: \"kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-kube-api-access-7qk99\") pod \"49cbfb85-a563-4265-828a-40ead144734b\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " Nov 03 20:27:39.044142 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.044112 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-socket\") pod \"49cbfb85-a563-4265-828a-40ead144734b\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " Nov 03 20:27:39.044333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.044194 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49cbfb85-a563-4265-828a-40ead144734b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cbfb85-a563-4265-828a-40ead144734b-istiod-ca-cert\") pod \"49cbfb85-a563-4265-828a-40ead144734b\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " Nov 03 20:27:39.044333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.044302 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-data\") pod \"49cbfb85-a563-4265-828a-40ead144734b\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " Nov 03 20:27:39.044490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.044408 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-istio-token\" (UniqueName: \"kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-istio-token\") pod \"49cbfb85-a563-4265-828a-40ead144734b\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " Nov 03 20:27:39.044490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.044473 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/49cbfb85-a563-4265-828a-40ead144734b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cbfb85-a563-4265-828a-40ead144734b-istio-podinfo\") pod \"49cbfb85-a563-4265-828a-40ead144734b\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " Nov 03 20:27:39.044640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.044603 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-certs\") pod \"49cbfb85-a563-4265-828a-40ead144734b\" (UID: \"49cbfb85-a563-4265-828a-40ead144734b\") " Nov 03 20:27:39.044994 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.044965 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-credential-socket" pod "49cbfb85-a563-4265-828a-40ead144734b" (UID: "49cbfb85-a563-4265-828a-40ead144734b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:39.045824 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.045801 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:39.045948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.045819 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-certs" pod "49cbfb85-a563-4265-828a-40ead144734b" (UID: "49cbfb85-a563-4265-828a-40ead144734b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:39.046081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.045973 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-data" pod "49cbfb85-a563-4265-828a-40ead144734b" (UID: "49cbfb85-a563-4265-828a-40ead144734b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:39.046081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.045871 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49cbfb85-a563-4265-828a-40ead144734b-istiod-ca-cert" pod "49cbfb85-a563-4265-828a-40ead144734b" (UID: "49cbfb85-a563-4265-828a-40ead144734b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:27:39.046081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.045930 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-socket" pod "49cbfb85-a563-4265-828a-40ead144734b" (UID: "49cbfb85-a563-4265-828a-40ead144734b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:39.048947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.048921 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-kube-api-access-7qk99" pod "49cbfb85-a563-4265-828a-40ead144734b" (UID: "49cbfb85-a563-4265-828a-40ead144734b"). InnerVolumeSpecName "kube-api-access-7qk99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:39.049473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.049439 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49cbfb85-a563-4265-828a-40ead144734b-istio-podinfo" pod "49cbfb85-a563-4265-828a-40ead144734b" (UID: "49cbfb85-a563-4265-828a-40ead144734b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:27:39.050376 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.050343 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-envoy" pod "49cbfb85-a563-4265-828a-40ead144734b" (UID: "49cbfb85-a563-4265-828a-40ead144734b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:27:39.050773 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.050727 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-istio-token" pod "49cbfb85-a563-4265-828a-40ead144734b" (UID: "49cbfb85-a563-4265-828a-40ead144734b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:27:39.148669 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.148575 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:39.148669 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.148612 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:39.148669 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.148626 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cbfb85-a563-4265-828a-40ead144734b-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:39.148669 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.148640 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:39.148669 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.148654 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:39.148669 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.148668 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qk99\" (UniqueName: \"kubernetes.io/projected/49cbfb85-a563-4265-828a-40ead144734b-kube-api-access-7qk99\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:39.149125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.148681 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cbfb85-a563-4265-828a-40ead144734b-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:39.149125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.148694 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cbfb85-a563-4265-828a-40ead144734b-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:27:39.212036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.212004 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22"] Nov 03 20:27:39.212170 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:27:39.212015 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d108227_cd53_4d58_9d65_1ff5fd6d1149.slice/crio-01c01b2edb357b087e77e9b9c16c8ac8ba57a96c5d58b54642f9dd4caa372a52 WatchSource:0}: Error finding container 01c01b2edb357b087e77e9b9c16c8ac8ba57a96c5d58b54642f9dd4caa372a52: Status 404 returned error can't find the container with id 01c01b2edb357b087e77e9b9c16c8ac8ba57a96c5d58b54642f9dd4caa372a52 Nov 03 20:27:39.215669 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.215628 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:39.215791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.215709 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:39.215791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.215750 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:27:39.368680 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.368648 4457 generic.go:356] "Generic (PLEG): container finished" podID="49cbfb85-a563-4265-828a-40ead144734b" containerID="f372bd35881b53c4a32db85199fc7910b04d16c8ae172f84953836040e55d9d8" exitCode=0 Nov 03 20:27:39.368872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.368723 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" event={"ID":"49cbfb85-a563-4265-828a-40ead144734b","Type":"ContainerDied","Data":"f372bd35881b53c4a32db85199fc7910b04d16c8ae172f84953836040e55d9d8"} Nov 03 20:27:39.368872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.368792 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" event={"ID":"49cbfb85-a563-4265-828a-40ead144734b","Type":"ContainerDied","Data":"860518f7694a8f2bfeed25639e79266b813934ad90dcee1049f830fe69776ba2"} Nov 03 20:27:39.368872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.368822 4457 scope.go:122] "RemoveContainer" containerID="f372bd35881b53c4a32db85199fc7910b04d16c8ae172f84953836040e55d9d8" Nov 03 20:27:39.369000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.368992 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf" Nov 03 20:27:39.379043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.379011 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" event={"ID":"8d108227-cd53-4d58-9d65-1ff5fd6d1149","Type":"ContainerStarted","Data":"644b7684c182052a788c5daf6815a0d00a650b751249db5a11830f58e16e8db2"} Nov 03 20:27:39.379197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.379074 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" event={"ID":"8d108227-cd53-4d58-9d65-1ff5fd6d1149","Type":"ContainerStarted","Data":"01c01b2edb357b087e77e9b9c16c8ac8ba57a96c5d58b54642f9dd4caa372a52"} Nov 03 20:27:39.397431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.394284 4457 scope.go:122] "RemoveContainer" containerID="fa891d3b4bad4521461ace4728816302f86a2b8ece0f8d5e701917cc39c88280" Nov 03 20:27:39.416903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.416618 4457 scope.go:122] "RemoveContainer" containerID="f372bd35881b53c4a32db85199fc7910b04d16c8ae172f84953836040e55d9d8" Nov 03 20:27:39.417359 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:27:39.417325 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f372bd35881b53c4a32db85199fc7910b04d16c8ae172f84953836040e55d9d8\": container with ID starting with f372bd35881b53c4a32db85199fc7910b04d16c8ae172f84953836040e55d9d8 not found: ID does not exist" containerID="f372bd35881b53c4a32db85199fc7910b04d16c8ae172f84953836040e55d9d8" Nov 03 20:27:39.417807 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.417369 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f372bd35881b53c4a32db85199fc7910b04d16c8ae172f84953836040e55d9d8"} err="failed to get container status \"f372bd35881b53c4a32db85199fc7910b04d16c8ae172f84953836040e55d9d8\": rpc error: code = NotFound desc = could not find container \"f372bd35881b53c4a32db85199fc7910b04d16c8ae172f84953836040e55d9d8\": container with ID starting with f372bd35881b53c4a32db85199fc7910b04d16c8ae172f84953836040e55d9d8 not found: ID does not exist" Nov 03 20:27:39.417892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.417805 4457 scope.go:122] "RemoveContainer" containerID="fa891d3b4bad4521461ace4728816302f86a2b8ece0f8d5e701917cc39c88280" Nov 03 20:27:39.418211 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:27:39.418183 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa891d3b4bad4521461ace4728816302f86a2b8ece0f8d5e701917cc39c88280\": container with ID starting with fa891d3b4bad4521461ace4728816302f86a2b8ece0f8d5e701917cc39c88280 not found: ID does not exist" containerID="fa891d3b4bad4521461ace4728816302f86a2b8ece0f8d5e701917cc39c88280" Nov 03 20:27:39.418295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.418220 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa891d3b4bad4521461ace4728816302f86a2b8ece0f8d5e701917cc39c88280"} err="failed to get container status \"fa891d3b4bad4521461ace4728816302f86a2b8ece0f8d5e701917cc39c88280\": rpc error: code = NotFound desc = could not find container \"fa891d3b4bad4521461ace4728816302f86a2b8ece0f8d5e701917cc39c88280\": container with ID starting with fa891d3b4bad4521461ace4728816302f86a2b8ece0f8d5e701917cc39c88280 not found: ID does not exist" Nov 03 20:27:39.420134 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.420064 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" podStartSLOduration=1.420052591 podStartE2EDuration="1.420052591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:39.411054235 +0000 UTC m=+1837.786650983" watchObservedRunningTime="2026-11-03 20:27:39.420052591 +0000 UTC m=+1837.795649344" Nov 03 20:27:39.432221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.432191 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf"] Nov 03 20:27:39.438742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.438715 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-ftckf"] Nov 03 20:27:39.619909 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.619875 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr"] Nov 03 20:27:39.621029 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.621002 4457 cpu_manager.go:414] "containerMap: removing container" podUID="49cbfb85-a563-4265-828a-40ead144734b" containerName="istio-proxy" Nov 03 20:27:39.621029 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.621028 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49cbfb85-a563-4265-828a-40ead144734b" containerName="istio-proxy" Nov 03 20:27:39.621209 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.621088 4457 cpu_manager.go:414] "containerMap: removing container" podUID="49cbfb85-a563-4265-828a-40ead144734b" containerName="istio-proxy" Nov 03 20:27:39.621209 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.621097 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49cbfb85-a563-4265-828a-40ead144734b" containerName="istio-proxy" Nov 03 20:27:39.621323 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.621305 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="49cbfb85-a563-4265-828a-40ead144734b" containerName="istio-proxy" Nov 03 20:27:39.621376 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.621334 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="49cbfb85-a563-4265-828a-40ead144734b" containerName="istio-proxy" Nov 03 20:27:39.626905 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.626874 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" Nov 03 20:27:39.636302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.636264 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr"] Nov 03 20:27:39.638144 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.638102 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c60d2a\"" Nov 03 20:27:39.638312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.638251 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c60d2a-dockercfg-6s2nk\"" Nov 03 20:27:39.673332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.673261 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx"] Nov 03 20:27:39.681500 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.680264 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" Nov 03 20:27:39.692493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.692463 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx"] Nov 03 20:27:39.860609 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.860550 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc31109-670b-421b-8d51-5bb50801cb55-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr\" (UID: \"fdc31109-670b-421b-8d51-5bb50801cb55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" Nov 03 20:27:39.860935 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.860910 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x98wj\" (UniqueName: \"kubernetes.io/projected/32bb3d1e-c3a2-401a-be41-dc09cebde931-kube-api-access-x98wj\") pod \"payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx\" (UID: \"32bb3d1e-c3a2-401a-be41-dc09cebde931\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" Nov 03 20:27:39.861309 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.861285 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lctvl\" (UniqueName: \"kubernetes.io/projected/fdc31109-670b-421b-8d51-5bb50801cb55-kube-api-access-lctvl\") pod \"payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr\" (UID: \"fdc31109-670b-421b-8d51-5bb50801cb55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" Nov 03 20:27:39.861629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.861609 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32bb3d1e-c3a2-401a-be41-dc09cebde931-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx\" (UID: \"32bb3d1e-c3a2-401a-be41-dc09cebde931\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" Nov 03 20:27:39.968523 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.968427 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc31109-670b-421b-8d51-5bb50801cb55-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr\" (UID: \"fdc31109-670b-421b-8d51-5bb50801cb55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" Nov 03 20:27:40.077647 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.077612 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98wj\" (UniqueName: \"kubernetes.io/projected/32bb3d1e-c3a2-401a-be41-dc09cebde931-kube-api-access-x98wj\") pod \"payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx\" (UID: \"32bb3d1e-c3a2-401a-be41-dc09cebde931\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" Nov 03 20:27:40.078780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.066350 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49cbfb85-a563-4265-828a-40ead144734b" path="/var/lib/kubelet/pods/49cbfb85-a563-4265-828a-40ead144734b/volumes" Nov 03 20:27:40.079019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.078987 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lctvl\" (UniqueName: \"kubernetes.io/projected/fdc31109-670b-421b-8d51-5bb50801cb55-kube-api-access-lctvl\") pod \"payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr\" (UID: \"fdc31109-670b-421b-8d51-5bb50801cb55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" Nov 03 20:27:40.079504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.079213 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32bb3d1e-c3a2-401a-be41-dc09cebde931-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx\" (UID: \"32bb3d1e-c3a2-401a-be41-dc09cebde931\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" Nov 03 20:27:40.080247 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.080219 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32bb3d1e-c3a2-401a-be41-dc09cebde931-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx\" (UID: \"32bb3d1e-c3a2-401a-be41-dc09cebde931\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" Nov 03 20:27:40.080474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.080455 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:40.081346 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:39.975413 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc31109-670b-421b-8d51-5bb50801cb55-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr\" (UID: \"fdc31109-670b-421b-8d51-5bb50801cb55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" Nov 03 20:27:40.127246 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.127195 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x98wj\" (UniqueName: \"kubernetes.io/projected/32bb3d1e-c3a2-401a-be41-dc09cebde931-kube-api-access-x98wj\") pod \"payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx\" (UID: \"32bb3d1e-c3a2-401a-be41-dc09cebde931\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" Nov 03 20:27:40.128273 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.128249 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lctvl\" (UniqueName: \"kubernetes.io/projected/fdc31109-670b-421b-8d51-5bb50801cb55-kube-api-access-lctvl\") pod \"payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr\" (UID: \"fdc31109-670b-421b-8d51-5bb50801cb55\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" Nov 03 20:27:40.145975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.145935 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" Nov 03 20:27:40.274698 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.274014 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" Nov 03 20:27:40.507523 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.507492 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx"] Nov 03 20:27:40.747165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.747123 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr"] Nov 03 20:27:40.886792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.884569 4457 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:27:40.886792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:40.884806 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" podUID="8d108227-cd53-4d58-9d65-1ff5fd6d1149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 03 20:27:41.417404 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:41.417353 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" event={"ID":"fdc31109-670b-421b-8d51-5bb50801cb55","Type":"ContainerStarted","Data":"3a773e6e04195e4c164086b1189bdf6d02acbd500778b02a017c36f41043a74c"} Nov 03 20:27:41.417767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:41.417730 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" event={"ID":"fdc31109-670b-421b-8d51-5bb50801cb55","Type":"ContainerStarted","Data":"d71fb928077b8c69e6f3e9a6b8bc997361868da88de8df2652855897b7879f83"} Nov 03 20:27:41.417896 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:41.417786 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" Nov 03 20:27:41.419766 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:41.419733 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" event={"ID":"32bb3d1e-c3a2-401a-be41-dc09cebde931","Type":"ContainerStarted","Data":"5c41416f32d196011791f63532c18ba3301ba31f14a12deb769d1414af6ed21c"} Nov 03 20:27:41.419890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:41.419783 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" event={"ID":"32bb3d1e-c3a2-401a-be41-dc09cebde931","Type":"ContainerStarted","Data":"37d0f1693ddf08a693f83764e4539d67c3d5685b13154161fdd75adb7ce55356"} Nov 03 20:27:41.419950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:41.419927 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" Nov 03 20:27:41.438277 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:41.438222 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" podStartSLOduration=2.438208849 podStartE2EDuration="2.438208849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:41.435510688 +0000 UTC m=+1839.811107433" watchObservedRunningTime="2026-11-03 20:27:41.438208849 +0000 UTC m=+1839.813805596" Nov 03 20:27:41.466257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:41.466186 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" podStartSLOduration=2.466164347 podStartE2EDuration="2.466164347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:41.454835403 +0000 UTC m=+1839.830432151" watchObservedRunningTime="2026-11-03 20:27:41.466164347 +0000 UTC m=+1839.841761098" Nov 03 20:27:41.884815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:41.884770 4457 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:27:41.885333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:41.884839 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" podUID="8d108227-cd53-4d58-9d65-1ff5fd6d1149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 03 20:27:42.889413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:42.884535 4457 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:27:42.889413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:42.884597 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" podUID="8d108227-cd53-4d58-9d65-1ff5fd6d1149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 03 20:27:43.428713 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:43.427718 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:43.496284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:43.496252 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:43.510258 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:43.510224 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:27:43.889492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:43.879475 4457 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:27:43.889492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:43.879547 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" podUID="8d108227-cd53-4d58-9d65-1ff5fd6d1149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 03 20:27:44.888296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:44.888197 4457 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:27:44.888296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:44.888277 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" podUID="8d108227-cd53-4d58-9d65-1ff5fd6d1149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 03 20:27:45.879200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:45.879150 4457 patch_prober.go:29] interesting pod/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" start-of-body= Nov 03 20:27:45.879742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:45.879221 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" podUID="8d108227-cd53-4d58-9d65-1ff5fd6d1149" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.199:15021/healthz/ready\": dial tcp 10.133.0.199:15021: connect: connection refused" Nov 03 20:27:46.886636 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:46.886584 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:46.887177 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:46.887156 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:46.887928 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:46.887909 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:27:49.348671 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.348636 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r"] Nov 03 20:27:49.355031 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.355008 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.380445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.375008 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a744eb\"/\"default-dockercfg-pvwwk\"" Nov 03 20:27:49.380445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.375111 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a744eb\"/\"kube-root-ca.crt\"" Nov 03 20:27:49.380445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.376264 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmp2l\" (UniqueName: \"kubernetes.io/projected/2a201136-0461-40e8-8388-ed9b7475223f-kube-api-access-fmp2l\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.380445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.376380 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-dshm\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.380445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.376556 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a201136-0461-40e8-8388-ed9b7475223f-tls-certs\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.380445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.376725 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-model-cache\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.380445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.376830 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-home\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.380445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.376895 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-tmp-dir\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.380445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.377254 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a744eb\"/\"test-model-a744eb-kserve-self-signed-certs\"" Nov 03 20:27:49.380445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.377351 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a744eb\"/\"openshift-service-ca.crt\"" Nov 03 20:27:49.380445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.380313 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r"] Nov 03 20:27:49.478361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.478324 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-tmp-dir\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.478361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.478371 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmp2l\" (UniqueName: \"kubernetes.io/projected/2a201136-0461-40e8-8388-ed9b7475223f-kube-api-access-fmp2l\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.478655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.478446 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-dshm\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.478655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.478556 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a201136-0461-40e8-8388-ed9b7475223f-tls-certs\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.478768 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.478691 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-model-cache\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.478824 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.478786 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-home\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.478880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.478833 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-tmp-dir\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.479256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.479206 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-model-cache\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.479804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.479777 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-home\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.481876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.481836 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-dshm\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.481997 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.481960 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a201136-0461-40e8-8388-ed9b7475223f-tls-certs\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.490625 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.490602 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmp2l\" (UniqueName: \"kubernetes.io/projected/2a201136-0461-40e8-8388-ed9b7475223f-kube-api-access-fmp2l\") pod \"test-model-a744eb-kserve-867c49fcc7-r2w4r\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:49.711461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:49.703800 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:50.139517 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:27:50.139483 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a201136_0461_40e8_8388_ed9b7475223f.slice/crio-2588f0d7b07866cd2d59c9e815a98795fde9632c8cd0d4494584748bdb6e967f WatchSource:0}: Error finding container 2588f0d7b07866cd2d59c9e815a98795fde9632c8cd0d4494584748bdb6e967f: Status 404 returned error can't find the container with id 2588f0d7b07866cd2d59c9e815a98795fde9632c8cd0d4494584748bdb6e967f Nov 03 20:27:50.141695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:50.141673 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r"] Nov 03 20:27:50.609866 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:50.609822 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" event={"ID":"2a201136-0461-40e8-8388-ed9b7475223f","Type":"ContainerStarted","Data":"38cfef2b0c4899cac208d8d5ef91bb86bbd7112588151a264fd2d939af8fb8b8"} Nov 03 20:27:50.610259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:50.609875 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" event={"ID":"2a201136-0461-40e8-8388-ed9b7475223f","Type":"ContainerStarted","Data":"2588f0d7b07866cd2d59c9e815a98795fde9632c8cd0d4494584748bdb6e967f"} Nov 03 20:27:50.631476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:50.631424 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" podStartSLOduration=1.63140923 podStartE2EDuration="1.63140923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:27:50.628702159 +0000 UTC m=+1849.004298908" watchObservedRunningTime="2026-11-03 20:27:50.63140923 +0000 UTC m=+1849.007005982" Nov 03 20:27:50.705698 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:50.705655 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:52.432234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:52.432204 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" Nov 03 20:27:52.432758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:52.432276 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" Nov 03 20:27:59.705447 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:59.705376 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:59.718543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:59.718513 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:27:59.748374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:27:59.748345 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:28:02.243501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.240040 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh"] Nov 03 20:28:02.248543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.248517 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.259890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.259864 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c60d2a-serving-cert\"" Nov 03 20:28:02.267659 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.267628 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh"] Nov 03 20:28:02.292081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.292045 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67ccec4-726a-4e35-a85f-baf6ff097782-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.292263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.292128 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckzf9\" (UniqueName: \"kubernetes.io/projected/c67ccec4-726a-4e35-a85f-baf6ff097782-kube-api-access-ckzf9\") pod \"maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.292344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.292286 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-maas-api-tls\") pod \"maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.292444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.292379 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-metrics-certs\") pod \"maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.394151 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.394093 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67ccec4-726a-4e35-a85f-baf6ff097782-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.394480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.394452 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckzf9\" (UniqueName: \"kubernetes.io/projected/c67ccec4-726a-4e35-a85f-baf6ff097782-kube-api-access-ckzf9\") pod \"maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.394692 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.394672 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-maas-api-tls\") pod \"maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.394867 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.394850 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-metrics-certs\") pod \"maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.399204 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.399178 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67ccec4-726a-4e35-a85f-baf6ff097782-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.399571 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.399535 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-metrics-certs\") pod \"maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.401479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.401457 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c60d2a-serving-cert\"" Nov 03 20:28:02.411115 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.411054 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckzf9\" (UniqueName: \"kubernetes.io/projected/c67ccec4-726a-4e35-a85f-baf6ff097782-kube-api-access-ckzf9\") pod \"maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.415732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.415697 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-maas-api-tls\") pod \"maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.577445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.577408 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:02.900376 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.900335 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh"] Nov 03 20:28:02.906073 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:28:02.906016 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67ccec4_726a_4e35_a85f_baf6ff097782.slice/crio-fd995ea5751ebdd97319c6294e70973f3679434d73fe9f1ae546cbb1a7d613e6 WatchSource:0}: Error finding container fd995ea5751ebdd97319c6294e70973f3679434d73fe9f1ae546cbb1a7d613e6: Status 404 returned error can't find the container with id fd995ea5751ebdd97319c6294e70973f3679434d73fe9f1ae546cbb1a7d613e6 Nov 03 20:28:02.910846 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:02.910821 4457 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:28:03.884945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:03.880165 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" event={"ID":"c67ccec4-726a-4e35-a85f-baf6ff097782","Type":"ContainerStarted","Data":"936fa7685dc62b50e0c192fb5c651a916c49bbcdfbb6e280ea6ec9398d568873"} Nov 03 20:28:03.884945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:03.880242 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" event={"ID":"c67ccec4-726a-4e35-a85f-baf6ff097782","Type":"ContainerStarted","Data":"fd995ea5751ebdd97319c6294e70973f3679434d73fe9f1ae546cbb1a7d613e6"} Nov 03 20:28:03.884945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:03.880290 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:03.901694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:03.901648 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" podStartSLOduration=1.3306877080000001 podStartE2EDuration="1.901633801s" totalImagesPullingTime="570.946093ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:03.899773771 +0000 UTC m=+1862.275370519" watchObservedRunningTime="2026-11-03 20:28:03.901633801 +0000 UTC m=+1862.277230553" Nov 03 20:28:06.149763 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:28:06.149721 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:28:07.094431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:07.094348 4457 scope.go:122] "RemoveContainer" containerID="097d6d84be7b51d32f9b41a9ddcfb798c44838b8478310ad9ac68bcebb554cc8" Nov 03 20:28:07.108370 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:07.108347 4457 scope.go:122] "RemoveContainer" containerID="57a0b7b15f2d6e16a194da47dd6698f8ddc70f00db16a53c884b8cfe852ce027" Nov 03 20:28:07.123072 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:07.123045 4457 scope.go:122] "RemoveContainer" containerID="049e63fdec50efe114a7882d06eddf85dc533a788c3eea86551bbbac62749c91" Nov 03 20:28:07.134364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:07.134337 4457 scope.go:122] "RemoveContainer" containerID="3e316fea2beaeb3010b82f51ea1d006251f9e94269731070754928b6d80c3322" Nov 03 20:28:10.914425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:10.913764 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:28:11.423399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.423329 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s"] Nov 03 20:28:11.489486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.489456 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s"] Nov 03 20:28:11.489853 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.489775 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" Nov 03 20:28:11.514731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.514696 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:28:11.515753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.515728 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:28:11.602940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.602549 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s\" (UID: \"55af785b-5cda-46fd-9f70-5eeb35039512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" Nov 03 20:28:11.602940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.602719 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2bqr\" (UniqueName: \"kubernetes.io/projected/55af785b-5cda-46fd-9f70-5eeb35039512-kube-api-access-h2bqr\") pod \"maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s\" (UID: \"55af785b-5cda-46fd-9f70-5eeb35039512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" Nov 03 20:28:11.602940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.602757 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s\" (UID: \"55af785b-5cda-46fd-9f70-5eeb35039512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" Nov 03 20:28:11.712170 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.712088 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s\" (UID: \"55af785b-5cda-46fd-9f70-5eeb35039512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" Nov 03 20:28:11.712515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.712488 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2bqr\" (UniqueName: \"kubernetes.io/projected/55af785b-5cda-46fd-9f70-5eeb35039512-kube-api-access-h2bqr\") pod \"maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s\" (UID: \"55af785b-5cda-46fd-9f70-5eeb35039512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" Nov 03 20:28:11.713474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.713447 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s\" (UID: \"55af785b-5cda-46fd-9f70-5eeb35039512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" Nov 03 20:28:11.714602 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.714580 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s\" (UID: \"55af785b-5cda-46fd-9f70-5eeb35039512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" Nov 03 20:28:11.728690 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.728661 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s\" (UID: \"55af785b-5cda-46fd-9f70-5eeb35039512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" Nov 03 20:28:11.733608 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.733566 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2bqr\" (UniqueName: \"kubernetes.io/projected/55af785b-5cda-46fd-9f70-5eeb35039512-kube-api-access-h2bqr\") pod \"maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s\" (UID: \"55af785b-5cda-46fd-9f70-5eeb35039512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" Nov 03 20:28:11.977462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:11.975899 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" Nov 03 20:28:12.388475 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:28:12.388440 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55af785b_5cda_46fd_9f70_5eeb35039512.slice/crio-ada12524caa1476f5a15ec0f0c6219347d760053acae202e02b58b741a0f9ea8 WatchSource:0}: Error finding container ada12524caa1476f5a15ec0f0c6219347d760053acae202e02b58b741a0f9ea8: Status 404 returned error can't find the container with id ada12524caa1476f5a15ec0f0c6219347d760053acae202e02b58b741a0f9ea8 Nov 03 20:28:12.408658 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:12.408628 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s"] Nov 03 20:28:13.099998 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:13.099945 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" event={"ID":"55af785b-5cda-46fd-9f70-5eeb35039512","Type":"ContainerStarted","Data":"7c833f359026c0662af271408141bff5d5201f2466a0b98d2082f84f5fa8f3fc"} Nov 03 20:28:13.100593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:13.100063 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" event={"ID":"55af785b-5cda-46fd-9f70-5eeb35039512","Type":"ContainerStarted","Data":"ada12524caa1476f5a15ec0f0c6219347d760053acae202e02b58b741a0f9ea8"} Nov 03 20:28:13.120793 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:13.120740 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" podStartSLOduration=2.120722296 podStartE2EDuration="2.120722296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:13.117496882 +0000 UTC m=+1871.493093630" watchObservedRunningTime="2026-11-03 20:28:13.120722296 +0000 UTC m=+1871.496319044" Nov 03 20:28:14.114479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:14.114436 4457 generic.go:356] "Generic (PLEG): container finished" podID="55af785b-5cda-46fd-9f70-5eeb35039512" containerID="7c833f359026c0662af271408141bff5d5201f2466a0b98d2082f84f5fa8f3fc" exitCode=0 Nov 03 20:28:14.114886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:14.114523 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" event={"ID":"55af785b-5cda-46fd-9f70-5eeb35039512","Type":"ContainerDied","Data":"7c833f359026c0662af271408141bff5d5201f2466a0b98d2082f84f5fa8f3fc"} Nov 03 20:28:15.410667 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:15.410637 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" Nov 03 20:28:15.435035 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:15.435004 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55af785b-5cda-46fd-9f70-5eeb35039512-kube-api-access-h2bqr\" (UniqueName: \"kubernetes.io/projected/55af785b-5cda-46fd-9f70-5eeb35039512-kube-api-access-h2bqr\") pod \"55af785b-5cda-46fd-9f70-5eeb35039512\" (UID: \"55af785b-5cda-46fd-9f70-5eeb35039512\") " Nov 03 20:28:15.435459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:15.435431 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-openshift-ca\") pod \"55af785b-5cda-46fd-9f70-5eeb35039512\" (UID: \"55af785b-5cda-46fd-9f70-5eeb35039512\") " Nov 03 20:28:15.435610 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:15.435478 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-xks-ca\") pod \"55af785b-5cda-46fd-9f70-5eeb35039512\" (UID: \"55af785b-5cda-46fd-9f70-5eeb35039512\") " Nov 03 20:28:15.436847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:15.436741 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-openshift-ca" pod "55af785b-5cda-46fd-9f70-5eeb35039512" (UID: "55af785b-5cda-46fd-9f70-5eeb35039512"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:28:15.438803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:15.438781 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55af785b-5cda-46fd-9f70-5eeb35039512-kube-api-access-h2bqr" pod "55af785b-5cda-46fd-9f70-5eeb35039512" (UID: "55af785b-5cda-46fd-9f70-5eeb35039512"). InnerVolumeSpecName "kube-api-access-h2bqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:15.439817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:15.439782 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-xks-ca" pod "55af785b-5cda-46fd-9f70-5eeb35039512" (UID: "55af785b-5cda-46fd-9f70-5eeb35039512"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:28:15.538124 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:15.538080 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2bqr\" (UniqueName: \"kubernetes.io/projected/55af785b-5cda-46fd-9f70-5eeb35039512-kube-api-access-h2bqr\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:15.538124 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:15.538116 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:15.538330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:15.538133 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55af785b-5cda-46fd-9f70-5eeb35039512-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:16.164898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:16.164859 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" event={"ID":"55af785b-5cda-46fd-9f70-5eeb35039512","Type":"ContainerDied","Data":"ada12524caa1476f5a15ec0f0c6219347d760053acae202e02b58b741a0f9ea8"} Nov 03 20:28:16.164898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:16.164890 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s" Nov 03 20:28:16.165460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:16.164919 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ada12524caa1476f5a15ec0f0c6219347d760053acae202e02b58b741a0f9ea8" Nov 03 20:28:20.247023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:20.246979 4457 generic.go:356] "Generic (PLEG): container finished" podID="7d097968-5008-4658-a534-fe81ba625490" containerID="4496f6ade23dd61b318645f2d4d985651b69df45c44dbf2265a9dec503169dfc" exitCode=0 Nov 03 20:28:20.247802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:20.247317 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" event={"ID":"7d097968-5008-4658-a534-fe81ba625490","Type":"ContainerDied","Data":"4496f6ade23dd61b318645f2d4d985651b69df45c44dbf2265a9dec503169dfc"} Nov 03 20:28:20.248258 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:20.248221 4457 scope.go:122] "RemoveContainer" containerID="4496f6ade23dd61b318645f2d4d985651b69df45c44dbf2265a9dec503169dfc" Nov 03 20:28:20.253495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:20.253442 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:20.253627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:20.253584 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:20.253691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:20.253647 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:21.259176 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:21.259139 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" event={"ID":"7d097968-5008-4658-a534-fe81ba625490","Type":"ContainerStarted","Data":"f22f5bf477b46a10362a194352a976a09502728dad81735d18865018fd129d48"} Nov 03 20:28:21.483416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:21.481101 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:28:21.483416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:21.481276 4457 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:28:21.483416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:21.481361 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 20:28:22.482414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:22.480890 4457 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:28:22.482414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:22.480953 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 20:28:23.486412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:23.480795 4457 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:28:23.486412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:23.480862 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 20:28:24.427416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:24.426460 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s"] Nov 03 20:28:24.454417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:24.453917 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5de824-2nf4s"] Nov 03 20:28:24.488414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:24.478815 4457 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:28:24.488414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:24.478884 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 20:28:24.532428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:24.522319 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:28:25.341940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.341907 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr"] Nov 03 20:28:25.342904 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.342885 4457 cpu_manager.go:414] "containerMap: removing container" podUID="55af785b-5cda-46fd-9f70-5eeb35039512" containerName="revoke-keys" Nov 03 20:28:25.342970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.342906 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55af785b-5cda-46fd-9f70-5eeb35039512" containerName="revoke-keys" Nov 03 20:28:25.343083 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.343073 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="55af785b-5cda-46fd-9f70-5eeb35039512" containerName="revoke-keys" Nov 03 20:28:25.346152 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.346132 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.354337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.353931 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c2c601-openshift-default-dockercfg-zq6pw\"" Nov 03 20:28:25.401608 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.401571 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr"] Nov 03 20:28:25.453139 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.453097 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-certs\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.453346 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.453228 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-envoy\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.453346 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.453283 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-istio-token\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.453346 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.453342 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf522b75-3374-41c3-8166-00e272080354-istiod-ca-cert\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.453509 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.453406 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-credential-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.453509 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.453484 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-data\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.453583 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.453541 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.453583 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.453559 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf522b75-3374-41c3-8166-00e272080354-istio-podinfo\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.453649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.453589 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqxzk\" (UniqueName: \"kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-kube-api-access-zqxzk\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.499524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.497866 4457 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:28:25.499524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.497930 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 20:28:25.555261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.555220 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-data\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.555480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.555354 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.563413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.559430 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf522b75-3374-41c3-8166-00e272080354-istio-podinfo\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.563413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.559509 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqxzk\" (UniqueName: \"kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-kube-api-access-zqxzk\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.563413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.559572 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-certs\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.563413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.559730 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-envoy\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.563413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.559739 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-data\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.563413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.559820 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-istio-token\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.563413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.559920 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf522b75-3374-41c3-8166-00e272080354-istiod-ca-cert\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.563413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.560010 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-credential-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.563413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.560520 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-credential-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.563413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.561104 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-certs\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.595095 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.564619 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-socket\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.595095 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.583610 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf522b75-3374-41c3-8166-00e272080354-istiod-ca-cert\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.613749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.613148 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-istio-token\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.613749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.613540 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-envoy\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.613749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.613688 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqxzk\" (UniqueName: \"kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-kube-api-access-zqxzk\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.614034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.613925 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf522b75-3374-41c3-8166-00e272080354-istio-podinfo\") pod \"e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.701779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.701736 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:25.789080 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:25.789043 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55af785b-5cda-46fd-9f70-5eeb35039512" path="/var/lib/kubelet/pods/55af785b-5cda-46fd-9f70-5eeb35039512/volumes" Nov 03 20:28:26.134640 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:28:26.134596 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf522b75_3374_41c3_8166_00e272080354.slice/crio-de0b0a633e8a5a661ede4642c634df525c373418e140c5bdadedc7744e8803d0 WatchSource:0}: Error finding container de0b0a633e8a5a661ede4642c634df525c373418e140c5bdadedc7744e8803d0: Status 404 returned error can't find the container with id de0b0a633e8a5a661ede4642c634df525c373418e140c5bdadedc7744e8803d0 Nov 03 20:28:26.135249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:26.135224 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr"] Nov 03 20:28:26.138543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:26.138506 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:26.138631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:26.138589 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:26.138675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:26.138630 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:28:26.482331 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:26.481833 4457 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:28:26.482331 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:26.481901 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 20:28:26.542265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:26.542225 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" event={"ID":"cf522b75-3374-41c3-8166-00e272080354","Type":"ContainerStarted","Data":"c81cbbfd8523c5fc065cc3d85c94b77cd21344601460bf8c42e06393bc025cde"} Nov 03 20:28:26.543011 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:26.542984 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" event={"ID":"cf522b75-3374-41c3-8166-00e272080354","Type":"ContainerStarted","Data":"de0b0a633e8a5a661ede4642c634df525c373418e140c5bdadedc7744e8803d0"} Nov 03 20:28:26.702628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:26.702546 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:27.482854 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:27.477795 4457 patch_prober.go:29] interesting pod/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" start-of-body= Nov 03 20:28:27.482854 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:27.477872 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.195:15021/healthz/ready\": dial tcp 10.133.0.195:15021: connect: connection refused" Nov 03 20:28:27.709949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:27.707958 4457 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 20:28:27.709949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:27.708018 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" podUID="cf522b75-3374-41c3-8166-00e272080354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 20:28:28.493711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:28.493679 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:28:28.496135 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:28.496113 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:28:28.520253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:28.520203 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" podStartSLOduration=3.520189454 podStartE2EDuration="3.520189454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:26.581767739 +0000 UTC m=+1884.957364487" watchObservedRunningTime="2026-11-03 20:28:28.520189454 +0000 UTC m=+1886.895786200" Nov 03 20:28:28.702570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:28.702534 4457 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 20:28:28.702905 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:28.702855 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" podUID="cf522b75-3374-41c3-8166-00e272080354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 20:28:29.702758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:29.702722 4457 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 20:28:29.703164 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:29.702808 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" podUID="cf522b75-3374-41c3-8166-00e272080354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 20:28:30.708412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:30.707939 4457 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 20:28:30.708412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:30.708013 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" podUID="cf522b75-3374-41c3-8166-00e272080354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 20:28:31.703129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:31.703087 4457 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 20:28:31.703307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:31.703167 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" podUID="cf522b75-3374-41c3-8166-00e272080354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 20:28:32.463413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:32.463343 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz"] Nov 03 20:28:32.507409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:32.482670 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" podUID="1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" containerName="istio-proxy" containerID="cri-o://4bb6d99741865b9cbfbeacd8f5a623be4dd0b016385e261544fb08e92037bf5d" gracePeriod=30 Nov 03 20:28:32.719476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:32.714034 4457 patch_prober.go:29] interesting pod/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" start-of-body= Nov 03 20:28:32.719476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:32.714094 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" podUID="cf522b75-3374-41c3-8166-00e272080354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.205:15021/healthz/ready\": dial tcp 10.133.0.205:15021: connect: connection refused" Nov 03 20:28:33.011083 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.010989 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z"] Nov 03 20:28:33.020163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.018430 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" Nov 03 20:28:33.026219 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.022560 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:28:33.026219 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.022713 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:28:33.026219 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.023731 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z"] Nov 03 20:28:33.033951 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.033640 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z\" (UID: \"ccc38539-b4bf-4211-a4a8-c2269f866600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" Nov 03 20:28:33.033951 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.033938 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25htv\" (UniqueName: \"kubernetes.io/projected/ccc38539-b4bf-4211-a4a8-c2269f866600-kube-api-access-25htv\") pod \"maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z\" (UID: \"ccc38539-b4bf-4211-a4a8-c2269f866600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" Nov 03 20:28:33.035701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.034846 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z\" (UID: \"ccc38539-b4bf-4211-a4a8-c2269f866600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" Nov 03 20:28:33.136346 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.136311 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25htv\" (UniqueName: \"kubernetes.io/projected/ccc38539-b4bf-4211-a4a8-c2269f866600-kube-api-access-25htv\") pod \"maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z\" (UID: \"ccc38539-b4bf-4211-a4a8-c2269f866600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" Nov 03 20:28:33.136563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.136465 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z\" (UID: \"ccc38539-b4bf-4211-a4a8-c2269f866600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" Nov 03 20:28:33.136563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.136540 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z\" (UID: \"ccc38539-b4bf-4211-a4a8-c2269f866600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" Nov 03 20:28:33.137350 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.137324 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z\" (UID: \"ccc38539-b4bf-4211-a4a8-c2269f866600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" Nov 03 20:28:33.139256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.139227 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z\" (UID: \"ccc38539-b4bf-4211-a4a8-c2269f866600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" Nov 03 20:28:33.148734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.148710 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25htv\" (UniqueName: \"kubernetes.io/projected/ccc38539-b4bf-4211-a4a8-c2269f866600-kube-api-access-25htv\") pod \"maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z\" (UID: \"ccc38539-b4bf-4211-a4a8-c2269f866600\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" Nov 03 20:28:33.343100 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.343067 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" Nov 03 20:28:33.364601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.364555 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh"] Nov 03 20:28:33.364870 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.364821 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" podUID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" containerName="istio-proxy" containerID="cri-o://1905035c712798dc2fbc6816f8719f2c4c1681b4e5260eb8e2bb08ca79a5d03a" gracePeriod=30 Nov 03 20:28:33.736093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.736051 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z"] Nov 03 20:28:33.742783 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.738263 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:33.764111 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:28:33.737071 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc38539_b4bf_4211_a4a8_c2269f866600.slice/crio-2c9d53da42942faaaa1ba9e527ebdf02f7614485efe7815cde20af8dd3907a84 WatchSource:0}: Error finding container 2c9d53da42942faaaa1ba9e527ebdf02f7614485efe7815cde20af8dd3907a84: Status 404 returned error can't find the container with id 2c9d53da42942faaaa1ba9e527ebdf02f7614485efe7815cde20af8dd3907a84 Nov 03 20:28:33.764309 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.764171 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:33.853038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.853009 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:28:33.928732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:33.928150 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" event={"ID":"ccc38539-b4bf-4211-a4a8-c2269f866600","Type":"ContainerStarted","Data":"2c9d53da42942faaaa1ba9e527ebdf02f7614485efe7815cde20af8dd3907a84"} Nov 03 20:28:34.943765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:34.943726 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" event={"ID":"ccc38539-b4bf-4211-a4a8-c2269f866600","Type":"ContainerStarted","Data":"450815db0b4c6a2538f2f108cdb8481a3f259203961a88a28cd7c5f9cbb1adbd"} Nov 03 20:28:34.963435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:34.963357 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" podStartSLOduration=2.963338268 podStartE2EDuration="2.963338268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:28:34.961383355 +0000 UTC m=+1893.336980102" watchObservedRunningTime="2026-11-03 20:28:34.963338268 +0000 UTC m=+1893.338935017" Nov 03 20:28:35.983892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:35.983842 4457 generic.go:356] "Generic (PLEG): container finished" podID="ccc38539-b4bf-4211-a4a8-c2269f866600" containerID="450815db0b4c6a2538f2f108cdb8481a3f259203961a88a28cd7c5f9cbb1adbd" exitCode=0 Nov 03 20:28:35.984743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:35.984706 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" event={"ID":"ccc38539-b4bf-4211-a4a8-c2269f866600","Type":"ContainerDied","Data":"450815db0b4c6a2538f2f108cdb8481a3f259203961a88a28cd7c5f9cbb1adbd"} Nov 03 20:28:36.155440 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:28:36.152681 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:28:37.229042 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.229016 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" Nov 03 20:28:37.336340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.336290 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-xks-ca\") pod \"ccc38539-b4bf-4211-a4a8-c2269f866600\" (UID: \"ccc38539-b4bf-4211-a4a8-c2269f866600\") " Nov 03 20:28:37.336543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.336516 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc38539-b4bf-4211-a4a8-c2269f866600-kube-api-access-25htv\" (UniqueName: \"kubernetes.io/projected/ccc38539-b4bf-4211-a4a8-c2269f866600-kube-api-access-25htv\") pod \"ccc38539-b4bf-4211-a4a8-c2269f866600\" (UID: \"ccc38539-b4bf-4211-a4a8-c2269f866600\") " Nov 03 20:28:37.336943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.336915 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-openshift-ca\") pod \"ccc38539-b4bf-4211-a4a8-c2269f866600\" (UID: \"ccc38539-b4bf-4211-a4a8-c2269f866600\") " Nov 03 20:28:37.338813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.338783 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-openshift-ca" pod "ccc38539-b4bf-4211-a4a8-c2269f866600" (UID: "ccc38539-b4bf-4211-a4a8-c2269f866600"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:28:37.346972 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.346932 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc38539-b4bf-4211-a4a8-c2269f866600-kube-api-access-25htv" pod "ccc38539-b4bf-4211-a4a8-c2269f866600" (UID: "ccc38539-b4bf-4211-a4a8-c2269f866600"). InnerVolumeSpecName "kube-api-access-25htv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:37.350564 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.350532 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-xks-ca" pod "ccc38539-b4bf-4211-a4a8-c2269f866600" (UID: "ccc38539-b4bf-4211-a4a8-c2269f866600"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:28:37.439910 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.439874 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:37.439910 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.439906 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc38539-b4bf-4211-a4a8-c2269f866600-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:37.439910 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.439916 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25htv\" (UniqueName: \"kubernetes.io/projected/ccc38539-b4bf-4211-a4a8-c2269f866600-kube-api-access-25htv\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:37.913530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.913451 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:28:37.979937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.979894 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-data\") pod \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " Nov 03 20:28:37.980711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.980515 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-data" pod "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" (UID: "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:37.980864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.980712 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-credential-socket\") pod \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " Nov 03 20:28:37.980864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.980750 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istiod-ca-cert\") pod \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " Nov 03 20:28:37.980864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.980815 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-token\" (UniqueName: \"kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-token\") pod \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " Nov 03 20:28:37.981044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.980877 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-certs\") pod \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " Nov 03 20:28:37.981044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.980940 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-kube-api-access-l5trk\" (UniqueName: \"kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-kube-api-access-l5trk\") pod \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " Nov 03 20:28:37.981044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.981038 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-socket\") pod \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " Nov 03 20:28:37.981201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.981107 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-envoy\") pod \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " Nov 03 20:28:37.982334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.981278 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-podinfo\") pod \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\" (UID: \"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf\") " Nov 03 20:28:37.982334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.982125 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:37.983034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.982730 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-credential-socket" pod "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" (UID: "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:37.983034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.982804 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-socket" pod "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" (UID: "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:37.983034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.982995 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istiod-ca-cert" pod "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" (UID: "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:28:37.983324 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.983299 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-certs" pod "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" (UID: "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:37.995609 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.995568 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-kube-api-access-l5trk" pod "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" (UID: "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf"). InnerVolumeSpecName "kube-api-access-l5trk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:37.995789 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.995704 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-token" pod "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" (UID: "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:37.997928 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:37.997899 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-envoy" pod "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" (UID: "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:38.000442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.000356 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-podinfo" pod "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" (UID: "1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:28:38.023784 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.023695 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" Nov 03 20:28:38.023784 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.023715 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z" event={"ID":"ccc38539-b4bf-4211-a4a8-c2269f866600","Type":"ContainerDied","Data":"2c9d53da42942faaaa1ba9e527ebdf02f7614485efe7815cde20af8dd3907a84"} Nov 03 20:28:38.023784 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.023790 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c9d53da42942faaaa1ba9e527ebdf02f7614485efe7815cde20af8dd3907a84" Nov 03 20:28:38.026886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.026854 4457 generic.go:356] "Generic (PLEG): container finished" podID="1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" containerID="4bb6d99741865b9cbfbeacd8f5a623be4dd0b016385e261544fb08e92037bf5d" exitCode=0 Nov 03 20:28:38.027057 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.026904 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" Nov 03 20:28:38.027057 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.026920 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" event={"ID":"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf","Type":"ContainerDied","Data":"4bb6d99741865b9cbfbeacd8f5a623be4dd0b016385e261544fb08e92037bf5d"} Nov 03 20:28:38.027057 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.026966 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz" event={"ID":"1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf","Type":"ContainerDied","Data":"e3bd40d7b9aa0580b71cd0c1525e3b725cc20cf4f3e765b72ebde98c34edb69a"} Nov 03 20:28:38.027057 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.026996 4457 scope.go:122] "RemoveContainer" containerID="4bb6d99741865b9cbfbeacd8f5a623be4dd0b016385e261544fb08e92037bf5d" Nov 03 20:28:38.056959 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.056933 4457 scope.go:122] "RemoveContainer" containerID="4bb6d99741865b9cbfbeacd8f5a623be4dd0b016385e261544fb08e92037bf5d" Nov 03 20:28:38.057354 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:28:38.057326 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb6d99741865b9cbfbeacd8f5a623be4dd0b016385e261544fb08e92037bf5d\": container with ID starting with 4bb6d99741865b9cbfbeacd8f5a623be4dd0b016385e261544fb08e92037bf5d not found: ID does not exist" containerID="4bb6d99741865b9cbfbeacd8f5a623be4dd0b016385e261544fb08e92037bf5d" Nov 03 20:28:38.058185 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.057367 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb6d99741865b9cbfbeacd8f5a623be4dd0b016385e261544fb08e92037bf5d"} err="failed to get container status \"4bb6d99741865b9cbfbeacd8f5a623be4dd0b016385e261544fb08e92037bf5d\": rpc error: code = NotFound desc = could not find container \"4bb6d99741865b9cbfbeacd8f5a623be4dd0b016385e261544fb08e92037bf5d\": container with ID starting with 4bb6d99741865b9cbfbeacd8f5a623be4dd0b016385e261544fb08e92037bf5d not found: ID does not exist" Nov 03 20:28:38.074967 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.074926 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz"] Nov 03 20:28:38.083715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.083683 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.083715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.083714 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.083974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.083731 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.083974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.083743 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.083974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.083755 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5trk\" (UniqueName: \"kubernetes.io/projected/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-kube-api-access-l5trk\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.083974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.083769 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.083974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.083808 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.083974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.083821 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.085598 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.085568 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5de824-openshift-default-6d9fc7cd7-9t5sz"] Nov 03 20:28:38.743821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.743757 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:28:38.811082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.811043 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-data\") pod \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " Nov 03 20:28:38.811257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.811150 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-certs\") pod \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " Nov 03 20:28:38.811257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.811175 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-kube-api-access-znlrn\" (UniqueName: \"kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-kube-api-access-znlrn\") pod \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " Nov 03 20:28:38.811257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.811225 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-envoy\") pod \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " Nov 03 20:28:38.811546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.811310 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-podinfo\") pod \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " Nov 03 20:28:38.811546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.811441 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-token\" (UniqueName: \"kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-token\") pod \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " Nov 03 20:28:38.811687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.811607 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istiod-ca-cert\") pod \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " Nov 03 20:28:38.811687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.811646 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-socket\") pod \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " Nov 03 20:28:38.811799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.811739 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-data" pod "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" (UID: "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:38.811799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.811760 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-credential-socket\") pod \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\" (UID: \"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0\") " Nov 03 20:28:38.813562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.813532 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.813671 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.813570 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-certs" pod "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" (UID: "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:38.813749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.813672 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-socket" pod "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" (UID: "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:38.814849 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.814827 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-credential-socket" pod "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" (UID: "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:38.814959 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.814944 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istiod-ca-cert" pod "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" (UID: "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:28:38.819623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.819593 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-kube-api-access-znlrn" pod "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" (UID: "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0"). InnerVolumeSpecName "kube-api-access-znlrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:38.819734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.819642 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-token" pod "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" (UID: "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:38.821240 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.821175 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-podinfo" pod "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" (UID: "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:28:38.821820 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.821792 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-envoy" pod "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" (UID: "91fd7838-627a-4d63-a0ba-ffbbe3b2afb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:28:38.915798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.915766 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.915798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.915796 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.915798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.915806 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znlrn\" (UniqueName: \"kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-kube-api-access-znlrn\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.916059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.915816 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.916059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.915825 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.916059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.915836 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.916059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.915849 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:38.916059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:38.915861 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:39.043157 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:39.043056 4457 generic.go:356] "Generic (PLEG): container finished" podID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" containerID="1905035c712798dc2fbc6816f8719f2c4c1681b4e5260eb8e2bb08ca79a5d03a" exitCode=0 Nov 03 20:28:39.043304 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:39.043188 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" event={"ID":"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0","Type":"ContainerDied","Data":"1905035c712798dc2fbc6816f8719f2c4c1681b4e5260eb8e2bb08ca79a5d03a"} Nov 03 20:28:39.043304 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:39.043239 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" event={"ID":"91fd7838-627a-4d63-a0ba-ffbbe3b2afb0","Type":"ContainerDied","Data":"8e62cd7cfb13aaabbd2ad3a0abe9742be5c39a8ea8b68feb81f27a2187c6556a"} Nov 03 20:28:39.043304 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:39.043269 4457 scope.go:122] "RemoveContainer" containerID="1905035c712798dc2fbc6816f8719f2c4c1681b4e5260eb8e2bb08ca79a5d03a" Nov 03 20:28:39.043488 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:39.043454 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh" Nov 03 20:28:39.062906 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:39.062871 4457 scope.go:122] "RemoveContainer" containerID="1905035c712798dc2fbc6816f8719f2c4c1681b4e5260eb8e2bb08ca79a5d03a" Nov 03 20:28:39.063253 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:28:39.063228 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1905035c712798dc2fbc6816f8719f2c4c1681b4e5260eb8e2bb08ca79a5d03a\": container with ID starting with 1905035c712798dc2fbc6816f8719f2c4c1681b4e5260eb8e2bb08ca79a5d03a not found: ID does not exist" containerID="1905035c712798dc2fbc6816f8719f2c4c1681b4e5260eb8e2bb08ca79a5d03a" Nov 03 20:28:39.063327 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:39.063265 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1905035c712798dc2fbc6816f8719f2c4c1681b4e5260eb8e2bb08ca79a5d03a"} err="failed to get container status \"1905035c712798dc2fbc6816f8719f2c4c1681b4e5260eb8e2bb08ca79a5d03a\": rpc error: code = NotFound desc = could not find container \"1905035c712798dc2fbc6816f8719f2c4c1681b4e5260eb8e2bb08ca79a5d03a\": container with ID starting with 1905035c712798dc2fbc6816f8719f2c4c1681b4e5260eb8e2bb08ca79a5d03a not found: ID does not exist" Nov 03 20:28:39.074930 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:39.074896 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh"] Nov 03 20:28:39.082630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:39.082601 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-x7frh"] Nov 03 20:28:39.768407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:39.766816 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" path="/var/lib/kubelet/pods/1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf/volumes" Nov 03 20:28:39.768407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:39.767727 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" path="/var/lib/kubelet/pods/91fd7838-627a-4d63-a0ba-ffbbe3b2afb0/volumes" Nov 03 20:28:43.248874 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:43.248838 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f"] Nov 03 20:28:48.250164 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:48.250039 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" podUID="037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf" containerName="payload-pre-processing" containerID="cri-o://9c4bb3151f01743d711ba215e08f99cb373b82f857ed12d7ebbbfb94707d7aa5" gracePeriod=25 Nov 03 20:28:48.580570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:48.580532 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" Nov 03 20:28:48.683877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:48.683837 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-plugins-config-volume\") pod \"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf\" (UID: \"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf\") " Nov 03 20:28:48.684046 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:48.683968 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-kube-api-access-99xq2\" (UniqueName: \"kubernetes.io/projected/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-kube-api-access-99xq2\") pod \"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf\" (UID: \"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf\") " Nov 03 20:28:48.684321 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:48.684289 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-plugins-config-volume" pod "037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf" (UID: "037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:28:48.684875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:48.684849 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:48.686257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:48.686233 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-kube-api-access-99xq2" pod "037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf" (UID: "037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf"). InnerVolumeSpecName "kube-api-access-99xq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:28:48.786579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:48.786546 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99xq2\" (UniqueName: \"kubernetes.io/projected/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf-kube-api-access-99xq2\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:28:49.182433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:49.182352 4457 generic.go:356] "Generic (PLEG): container finished" podID="037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf" containerID="9c4bb3151f01743d711ba215e08f99cb373b82f857ed12d7ebbbfb94707d7aa5" exitCode=0 Nov 03 20:28:49.182433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:49.182421 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" Nov 03 20:28:49.182653 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:49.182459 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" event={"ID":"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf","Type":"ContainerDied","Data":"9c4bb3151f01743d711ba215e08f99cb373b82f857ed12d7ebbbfb94707d7aa5"} Nov 03 20:28:49.182653 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:49.182512 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f" event={"ID":"037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf","Type":"ContainerDied","Data":"48a5e6e6797cf534f399e04d5a0b36562a046f2c9bc9e8fe1e5d75732c562212"} Nov 03 20:28:49.182653 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:49.182535 4457 scope.go:122] "RemoveContainer" containerID="9c4bb3151f01743d711ba215e08f99cb373b82f857ed12d7ebbbfb94707d7aa5" Nov 03 20:28:49.202559 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:49.202524 4457 scope.go:122] "RemoveContainer" containerID="9c4bb3151f01743d711ba215e08f99cb373b82f857ed12d7ebbbfb94707d7aa5" Nov 03 20:28:49.204547 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:28:49.204518 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4bb3151f01743d711ba215e08f99cb373b82f857ed12d7ebbbfb94707d7aa5\": container with ID starting with 9c4bb3151f01743d711ba215e08f99cb373b82f857ed12d7ebbbfb94707d7aa5 not found: ID does not exist" containerID="9c4bb3151f01743d711ba215e08f99cb373b82f857ed12d7ebbbfb94707d7aa5" Nov 03 20:28:49.204661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:49.204561 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4bb3151f01743d711ba215e08f99cb373b82f857ed12d7ebbbfb94707d7aa5"} err="failed to get container status \"9c4bb3151f01743d711ba215e08f99cb373b82f857ed12d7ebbbfb94707d7aa5\": rpc error: code = NotFound desc = could not find container \"9c4bb3151f01743d711ba215e08f99cb373b82f857ed12d7ebbbfb94707d7aa5\": container with ID starting with 9c4bb3151f01743d711ba215e08f99cb373b82f857ed12d7ebbbfb94707d7aa5 not found: ID does not exist" Nov 03 20:28:49.221244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:49.221204 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f"] Nov 03 20:28:49.245907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:49.245869 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d9f082-6bd859c959-fcc4f"] Nov 03 20:28:49.767509 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:49.767056 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf" path="/var/lib/kubelet/pods/037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf/volumes" Nov 03 20:28:59.586860 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:59.584596 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z"] Nov 03 20:28:59.604083 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:59.604041 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d9f082-r5c7z"] Nov 03 20:28:59.818849 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:28:59.807849 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc38539-b4bf-4211-a4a8-c2269f866600" path="/var/lib/kubelet/pods/ccc38539-b4bf-4211-a4a8-c2269f866600/volumes" Nov 03 20:29:06.260568 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:29:06.260519 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:29:06.640123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:06.640091 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q"] Nov 03 20:29:06.640577 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:06.640543 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" containerID="cri-o://f8319497a62ab056cb6747a32c5b3b8e957d7b3d7960535c5d5511f7c438bd5a" gracePeriod=30 Nov 03 20:29:07.202711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.202677 4457 scope.go:122] "RemoveContainer" containerID="8397124c8293e5020d581883c0b390180d79f2393be929d28f2deb06e331f4e9" Nov 03 20:29:07.214515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.214490 4457 scope.go:122] "RemoveContainer" containerID="ae44e10270bce3c2c7a3fbe2690b29b4e8ab2b9d32d6c44c40708c8b87ea96eb" Nov 03 20:29:07.290130 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.290100 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f"] Nov 03 20:29:07.291120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.291102 4457 cpu_manager.go:414] "containerMap: removing container" podUID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" containerName="istio-proxy" Nov 03 20:29:07.291175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.291122 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" containerName="istio-proxy" Nov 03 20:29:07.291175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.291155 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ccc38539-b4bf-4211-a4a8-c2269f866600" containerName="revoke-keys" Nov 03 20:29:07.291175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.291160 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc38539-b4bf-4211-a4a8-c2269f866600" containerName="revoke-keys" Nov 03 20:29:07.291175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.291170 4457 cpu_manager.go:414] "containerMap: removing container" podUID="037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf" containerName="payload-pre-processing" Nov 03 20:29:07.291309 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.291177 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf" containerName="payload-pre-processing" Nov 03 20:29:07.291309 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.291186 4457 cpu_manager.go:414] "containerMap: removing container" podUID="1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" containerName="istio-proxy" Nov 03 20:29:07.291309 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.291190 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" containerName="istio-proxy" Nov 03 20:29:07.291430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.291319 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc38539-b4bf-4211-a4a8-c2269f866600" containerName="revoke-keys" Nov 03 20:29:07.291430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.291329 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="037b282c-3c0b-4fa5-bcdc-eb0f4aa1babf" containerName="payload-pre-processing" Nov 03 20:29:07.291430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.291337 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd50fe3-bca7-4aa8-8ba6-75226b0adfdf" containerName="istio-proxy" Nov 03 20:29:07.291430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.291353 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fd7838-627a-4d63-a0ba-ffbbe3b2afb0" containerName="istio-proxy" Nov 03 20:29:07.297245 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.297222 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" Nov 03 20:29:07.302180 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.302132 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:29:07.302743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.302721 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:29:07.304644 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.304620 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f"] Nov 03 20:29:07.402041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.401995 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv9sn\" (UniqueName: \"kubernetes.io/projected/1787b9bc-7db1-4195-b51e-fa5b705f3124-kube-api-access-wv9sn\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f\" (UID: \"1787b9bc-7db1-4195-b51e-fa5b705f3124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" Nov 03 20:29:07.402224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.402170 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f\" (UID: \"1787b9bc-7db1-4195-b51e-fa5b705f3124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" Nov 03 20:29:07.402484 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.402464 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f\" (UID: \"1787b9bc-7db1-4195-b51e-fa5b705f3124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" Nov 03 20:29:07.504174 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.504075 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f\" (UID: \"1787b9bc-7db1-4195-b51e-fa5b705f3124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" Nov 03 20:29:07.504361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.504215 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv9sn\" (UniqueName: \"kubernetes.io/projected/1787b9bc-7db1-4195-b51e-fa5b705f3124-kube-api-access-wv9sn\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f\" (UID: \"1787b9bc-7db1-4195-b51e-fa5b705f3124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" Nov 03 20:29:07.504361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.504277 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f\" (UID: \"1787b9bc-7db1-4195-b51e-fa5b705f3124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" Nov 03 20:29:07.505015 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.504984 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f\" (UID: \"1787b9bc-7db1-4195-b51e-fa5b705f3124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" Nov 03 20:29:07.507179 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.507143 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f\" (UID: \"1787b9bc-7db1-4195-b51e-fa5b705f3124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" Nov 03 20:29:07.516879 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.516854 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv9sn\" (UniqueName: \"kubernetes.io/projected/1787b9bc-7db1-4195-b51e-fa5b705f3124-kube-api-access-wv9sn\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f\" (UID: \"1787b9bc-7db1-4195-b51e-fa5b705f3124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" Nov 03 20:29:07.615767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.615732 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" Nov 03 20:29:07.920847 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:29:07.920812 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1787b9bc_7db1_4195_b51e_fa5b705f3124.slice/crio-2baa10b601a239db29eadb082117419c936b72e6d7aedf5655fec8309302f317 WatchSource:0}: Error finding container 2baa10b601a239db29eadb082117419c936b72e6d7aedf5655fec8309302f317: Status 404 returned error can't find the container with id 2baa10b601a239db29eadb082117419c936b72e6d7aedf5655fec8309302f317 Nov 03 20:29:07.922334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:07.922308 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f"] Nov 03 20:29:08.438803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:08.438764 4457 patch_prober.go:29] interesting pod/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:29:08.439202 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:08.438827 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:29:08.452805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:08.452774 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" event={"ID":"1787b9bc-7db1-4195-b51e-fa5b705f3124","Type":"ContainerStarted","Data":"2baa10b601a239db29eadb082117419c936b72e6d7aedf5655fec8309302f317"} Nov 03 20:29:09.465199 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:09.465101 4457 generic.go:356] "Generic (PLEG): container finished" podID="1787b9bc-7db1-4195-b51e-fa5b705f3124" containerID="bb1a40645b3a4a7a180c45273c1cfd6d697818af9e4c6c313f58193a09fcfcdb" exitCode=0 Nov 03 20:29:09.465691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:09.465233 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" event={"ID":"1787b9bc-7db1-4195-b51e-fa5b705f3124","Type":"ContainerDied","Data":"bb1a40645b3a4a7a180c45273c1cfd6d697818af9e4c6c313f58193a09fcfcdb"} Nov 03 20:29:10.725752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:10.725724 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" Nov 03 20:29:10.879985 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:10.879950 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-openshift-ca\") pod \"1787b9bc-7db1-4195-b51e-fa5b705f3124\" (UID: \"1787b9bc-7db1-4195-b51e-fa5b705f3124\") " Nov 03 20:29:10.880299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:10.880279 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1787b9bc-7db1-4195-b51e-fa5b705f3124-kube-api-access-wv9sn\" (UniqueName: \"kubernetes.io/projected/1787b9bc-7db1-4195-b51e-fa5b705f3124-kube-api-access-wv9sn\") pod \"1787b9bc-7db1-4195-b51e-fa5b705f3124\" (UID: \"1787b9bc-7db1-4195-b51e-fa5b705f3124\") " Nov 03 20:29:10.880465 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:10.880368 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-openshift-ca" pod "1787b9bc-7db1-4195-b51e-fa5b705f3124" (UID: "1787b9bc-7db1-4195-b51e-fa5b705f3124"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:10.880465 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:10.880397 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-xks-ca\") pod \"1787b9bc-7db1-4195-b51e-fa5b705f3124\" (UID: \"1787b9bc-7db1-4195-b51e-fa5b705f3124\") " Nov 03 20:29:10.881274 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:10.881248 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:10.882673 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:10.882648 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-xks-ca" pod "1787b9bc-7db1-4195-b51e-fa5b705f3124" (UID: "1787b9bc-7db1-4195-b51e-fa5b705f3124"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:29:10.883014 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:10.882992 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1787b9bc-7db1-4195-b51e-fa5b705f3124-kube-api-access-wv9sn" pod "1787b9bc-7db1-4195-b51e-fa5b705f3124" (UID: "1787b9bc-7db1-4195-b51e-fa5b705f3124"). InnerVolumeSpecName "kube-api-access-wv9sn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:10.984019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:10.983980 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv9sn\" (UniqueName: \"kubernetes.io/projected/1787b9bc-7db1-4195-b51e-fa5b705f3124-kube-api-access-wv9sn\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:10.984019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:10.984021 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1787b9bc-7db1-4195-b51e-fa5b705f3124-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:11.491775 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:11.491738 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" Nov 03 20:29:11.491775 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:11.491752 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f" event={"ID":"1787b9bc-7db1-4195-b51e-fa5b705f3124","Type":"ContainerDied","Data":"2baa10b601a239db29eadb082117419c936b72e6d7aedf5655fec8309302f317"} Nov 03 20:29:11.492009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:11.491801 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2baa10b601a239db29eadb082117419c936b72e6d7aedf5655fec8309302f317" Nov 03 20:29:11.981090 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:11.981058 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:29:12.107007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.106965 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istiod-ca-cert\") pod \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " Nov 03 20:29:12.107590 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.107569 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-kube-api-access-dnqpt\" (UniqueName: \"kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-kube-api-access-dnqpt\") pod \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " Nov 03 20:29:12.107715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.107519 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istiod-ca-cert" pod "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" (UID: "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:12.107715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.107686 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-envoy\") pod \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " Nov 03 20:29:12.108068 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.108043 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-podinfo\") pod \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " Nov 03 20:29:12.108167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.108102 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-data\") pod \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " Nov 03 20:29:12.108228 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.108190 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-credential-socket\") pod \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " Nov 03 20:29:12.108299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.108287 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-token\" (UniqueName: \"kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-token\") pod \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " Nov 03 20:29:12.108646 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.108358 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-certs\") pod \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " Nov 03 20:29:12.108646 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.108464 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-data" pod "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" (UID: "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:12.108646 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.108500 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-socket\") pod \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\" (UID: \"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2\") " Nov 03 20:29:12.108855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.108653 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-credential-socket" pod "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" (UID: "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:12.109656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.109059 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-certs" pod "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" (UID: "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:12.109656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.109451 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-socket" pod "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" (UID: "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:12.109656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.109561 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.109656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.109585 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.109656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.109601 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.109656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.109624 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.110363 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.110341 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-envoy" pod "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" (UID: "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:12.111625 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.111593 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-kube-api-access-dnqpt" pod "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" (UID: "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2"). InnerVolumeSpecName "kube-api-access-dnqpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:12.111715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.111641 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-token" pod "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" (UID: "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:12.111715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.111658 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-podinfo" pod "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" (UID: "d0386779-b087-4c1b-9c9e-0a38d3e9a8d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:12.214560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.214519 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnqpt\" (UniqueName: \"kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-kube-api-access-dnqpt\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.214560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.214562 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.214792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.214579 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.214792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.214592 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.214792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.214607 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:12.528541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.528440 4457 generic.go:356] "Generic (PLEG): container finished" podID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerID="f8319497a62ab056cb6747a32c5b3b8e957d7b3d7960535c5d5511f7c438bd5a" exitCode=0 Nov 03 20:29:12.528721 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.528677 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" event={"ID":"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2","Type":"ContainerDied","Data":"f8319497a62ab056cb6747a32c5b3b8e957d7b3d7960535c5d5511f7c438bd5a"} Nov 03 20:29:12.528793 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.528773 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" event={"ID":"d0386779-b087-4c1b-9c9e-0a38d3e9a8d2","Type":"ContainerDied","Data":"5d5d65fa3dd5d21620af012e47ac1660e5a150a620d02fbd9b7833a952be41d9"} Nov 03 20:29:12.528848 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.528822 4457 scope.go:122] "RemoveContainer" containerID="f8319497a62ab056cb6747a32c5b3b8e957d7b3d7960535c5d5511f7c438bd5a" Nov 03 20:29:12.529068 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.529049 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q" Nov 03 20:29:12.547530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.547501 4457 scope.go:122] "RemoveContainer" containerID="372a03fdf78fdf92ee8b8db1f9918d1ac307245577fe1431ea379afc88676a45" Nov 03 20:29:12.565686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.565574 4457 scope.go:122] "RemoveContainer" containerID="f8319497a62ab056cb6747a32c5b3b8e957d7b3d7960535c5d5511f7c438bd5a" Nov 03 20:29:12.565969 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:29:12.565940 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8319497a62ab056cb6747a32c5b3b8e957d7b3d7960535c5d5511f7c438bd5a\": container with ID starting with f8319497a62ab056cb6747a32c5b3b8e957d7b3d7960535c5d5511f7c438bd5a not found: ID does not exist" containerID="f8319497a62ab056cb6747a32c5b3b8e957d7b3d7960535c5d5511f7c438bd5a" Nov 03 20:29:12.566060 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.565982 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8319497a62ab056cb6747a32c5b3b8e957d7b3d7960535c5d5511f7c438bd5a"} err="failed to get container status \"f8319497a62ab056cb6747a32c5b3b8e957d7b3d7960535c5d5511f7c438bd5a\": rpc error: code = NotFound desc = could not find container \"f8319497a62ab056cb6747a32c5b3b8e957d7b3d7960535c5d5511f7c438bd5a\": container with ID starting with f8319497a62ab056cb6747a32c5b3b8e957d7b3d7960535c5d5511f7c438bd5a not found: ID does not exist" Nov 03 20:29:12.566060 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.566006 4457 scope.go:122] "RemoveContainer" containerID="372a03fdf78fdf92ee8b8db1f9918d1ac307245577fe1431ea379afc88676a45" Nov 03 20:29:12.568442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.568381 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q"] Nov 03 20:29:12.569825 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:29:12.569799 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"372a03fdf78fdf92ee8b8db1f9918d1ac307245577fe1431ea379afc88676a45\": container with ID starting with 372a03fdf78fdf92ee8b8db1f9918d1ac307245577fe1431ea379afc88676a45 not found: ID does not exist" containerID="372a03fdf78fdf92ee8b8db1f9918d1ac307245577fe1431ea379afc88676a45" Nov 03 20:29:12.570046 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.569998 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"372a03fdf78fdf92ee8b8db1f9918d1ac307245577fe1431ea379afc88676a45"} err="failed to get container status \"372a03fdf78fdf92ee8b8db1f9918d1ac307245577fe1431ea379afc88676a45\": rpc error: code = NotFound desc = could not find container \"372a03fdf78fdf92ee8b8db1f9918d1ac307245577fe1431ea379afc88676a45\": container with ID starting with 372a03fdf78fdf92ee8b8db1f9918d1ac307245577fe1431ea379afc88676a45 not found: ID does not exist" Nov 03 20:29:12.577558 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:12.577525 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d9f082-openshift-default-7646ddc-rtx4q"] Nov 03 20:29:12.610200 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:29:12.610168 4457 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0386779_b087_4c1b_9c9e_0a38d3e9a8d2.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0386779_b087_4c1b_9c9e_0a38d3e9a8d2.slice/crio-5d5d65fa3dd5d21620af012e47ac1660e5a150a620d02fbd9b7833a952be41d9\": RecentStats: unable to find data in memory cache]" Nov 03 20:29:13.762895 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:13.762859 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" path="/var/lib/kubelet/pods/d0386779-b087-4c1b-9c9e-0a38d3e9a8d2/volumes" Nov 03 20:29:17.446398 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:17.446290 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7"] Nov 03 20:29:18.404081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:18.404049 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw"] Nov 03 20:29:18.404285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:18.404263 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" podUID="27621a94-aeca-461d-bf48-cddfb8c0ef53" containerName="istio-proxy" containerID="cri-o://860c65d54d7a53b68c663a8edd9a28c488a93d082eba2cc085eca2a697d3a7a8" gracePeriod=30 Nov 03 20:29:22.447654 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:22.447611 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" podUID="cc705cd8-f6b5-4907-a90c-39f0c779f227" containerName="payload-pre-processing" containerID="cri-o://938c78e5fd248904c6b7d217fee2fa51bb64397dd0ddf1f51bcd37ccd6286871" gracePeriod=25 Nov 03 20:29:22.736899 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:22.736862 4457 generic.go:356] "Generic (PLEG): container finished" podID="cc705cd8-f6b5-4907-a90c-39f0c779f227" containerID="938c78e5fd248904c6b7d217fee2fa51bb64397dd0ddf1f51bcd37ccd6286871" exitCode=0 Nov 03 20:29:22.737181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:22.737156 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" event={"ID":"cc705cd8-f6b5-4907-a90c-39f0c779f227","Type":"ContainerDied","Data":"938c78e5fd248904c6b7d217fee2fa51bb64397dd0ddf1f51bcd37ccd6286871"} Nov 03 20:29:22.737347 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:22.737325 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" event={"ID":"cc705cd8-f6b5-4907-a90c-39f0c779f227","Type":"ContainerDied","Data":"a444961af2e8128cffecddd4471f108818d7802a35a7d2d72aab8e9bb79a6881"} Nov 03 20:29:22.737501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:22.737476 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a444961af2e8128cffecddd4471f108818d7802a35a7d2d72aab8e9bb79a6881" Nov 03 20:29:22.777923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:22.777874 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" Nov 03 20:29:22.870801 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:22.870765 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc705cd8-f6b5-4907-a90c-39f0c779f227-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc705cd8-f6b5-4907-a90c-39f0c779f227-plugins-config-volume\") pod \"cc705cd8-f6b5-4907-a90c-39f0c779f227\" (UID: \"cc705cd8-f6b5-4907-a90c-39f0c779f227\") " Nov 03 20:29:22.871059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:22.870914 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc705cd8-f6b5-4907-a90c-39f0c779f227-kube-api-access-j76bm\" (UniqueName: \"kubernetes.io/projected/cc705cd8-f6b5-4907-a90c-39f0c779f227-kube-api-access-j76bm\") pod \"cc705cd8-f6b5-4907-a90c-39f0c779f227\" (UID: \"cc705cd8-f6b5-4907-a90c-39f0c779f227\") " Nov 03 20:29:22.871251 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:22.871218 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc705cd8-f6b5-4907-a90c-39f0c779f227-plugins-config-volume" pod "cc705cd8-f6b5-4907-a90c-39f0c779f227" (UID: "cc705cd8-f6b5-4907-a90c-39f0c779f227"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:22.871903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:22.871870 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc705cd8-f6b5-4907-a90c-39f0c779f227-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:22.873416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:22.873371 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc705cd8-f6b5-4907-a90c-39f0c779f227-kube-api-access-j76bm" pod "cc705cd8-f6b5-4907-a90c-39f0c779f227" (UID: "cc705cd8-f6b5-4907-a90c-39f0c779f227"). InnerVolumeSpecName "kube-api-access-j76bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:22.973632 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:22.973546 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j76bm\" (UniqueName: \"kubernetes.io/projected/cc705cd8-f6b5-4907-a90c-39f0c779f227-kube-api-access-j76bm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.808009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.805124 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:29:23.818378 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.817946 4457 generic.go:356] "Generic (PLEG): container finished" podID="27621a94-aeca-461d-bf48-cddfb8c0ef53" containerID="860c65d54d7a53b68c663a8edd9a28c488a93d082eba2cc085eca2a697d3a7a8" exitCode=0 Nov 03 20:29:23.818378 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.818134 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7" Nov 03 20:29:23.824034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.823990 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" event={"ID":"27621a94-aeca-461d-bf48-cddfb8c0ef53","Type":"ContainerDied","Data":"860c65d54d7a53b68c663a8edd9a28c488a93d082eba2cc085eca2a697d3a7a8"} Nov 03 20:29:23.824206 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.824067 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" event={"ID":"27621a94-aeca-461d-bf48-cddfb8c0ef53","Type":"ContainerDied","Data":"8af2e3541232d886576f43c9eb06b82ac7942806cf07016136687b9b1c7ed513"} Nov 03 20:29:23.824206 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.824099 4457 scope.go:122] "RemoveContainer" containerID="860c65d54d7a53b68c663a8edd9a28c488a93d082eba2cc085eca2a697d3a7a8" Nov 03 20:29:23.865524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.865502 4457 scope.go:122] "RemoveContainer" containerID="860c65d54d7a53b68c663a8edd9a28c488a93d082eba2cc085eca2a697d3a7a8" Nov 03 20:29:23.865920 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:29:23.865891 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"860c65d54d7a53b68c663a8edd9a28c488a93d082eba2cc085eca2a697d3a7a8\": container with ID starting with 860c65d54d7a53b68c663a8edd9a28c488a93d082eba2cc085eca2a697d3a7a8 not found: ID does not exist" containerID="860c65d54d7a53b68c663a8edd9a28c488a93d082eba2cc085eca2a697d3a7a8" Nov 03 20:29:23.866043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.865931 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"860c65d54d7a53b68c663a8edd9a28c488a93d082eba2cc085eca2a697d3a7a8"} err="failed to get container status \"860c65d54d7a53b68c663a8edd9a28c488a93d082eba2cc085eca2a697d3a7a8\": rpc error: code = NotFound desc = could not find container \"860c65d54d7a53b68c663a8edd9a28c488a93d082eba2cc085eca2a697d3a7a8\": container with ID starting with 860c65d54d7a53b68c663a8edd9a28c488a93d082eba2cc085eca2a697d3a7a8 not found: ID does not exist" Nov 03 20:29:23.888132 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.888095 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7"] Nov 03 20:29:23.893528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.893496 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-7fd1a7-59c7dd89f9-59vx7"] Nov 03 20:29:23.895443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.895412 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-data\") pod \"27621a94-aeca-461d-bf48-cddfb8c0ef53\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " Nov 03 20:29:23.895568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.895551 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-certs\") pod \"27621a94-aeca-461d-bf48-cddfb8c0ef53\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " Nov 03 20:29:23.895623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.895585 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-envoy\") pod \"27621a94-aeca-461d-bf48-cddfb8c0ef53\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " Nov 03 20:29:23.895681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.895635 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-data" pod "27621a94-aeca-461d-bf48-cddfb8c0ef53" (UID: "27621a94-aeca-461d-bf48-cddfb8c0ef53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:23.895750 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.895698 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-podinfo\") pod \"27621a94-aeca-461d-bf48-cddfb8c0ef53\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " Nov 03 20:29:23.895807 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.895746 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-credential-socket\") pod \"27621a94-aeca-461d-bf48-cddfb8c0ef53\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " Nov 03 20:29:23.895882 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.895860 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-certs" pod "27621a94-aeca-461d-bf48-cddfb8c0ef53" (UID: "27621a94-aeca-461d-bf48-cddfb8c0ef53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:23.895946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.895915 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27621a94-aeca-461d-bf48-cddfb8c0ef53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27621a94-aeca-461d-bf48-cddfb8c0ef53-istiod-ca-cert\") pod \"27621a94-aeca-461d-bf48-cddfb8c0ef53\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " Nov 03 20:29:23.896063 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.896047 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-socket\") pod \"27621a94-aeca-461d-bf48-cddfb8c0ef53\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " Nov 03 20:29:23.896134 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.896063 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-credential-socket" pod "27621a94-aeca-461d-bf48-cddfb8c0ef53" (UID: "27621a94-aeca-461d-bf48-cddfb8c0ef53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:23.896195 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.896178 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-kube-api-access-q6jd9\" (UniqueName: \"kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-kube-api-access-q6jd9\") pod \"27621a94-aeca-461d-bf48-cddfb8c0ef53\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " Nov 03 20:29:23.896252 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.896220 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-token\" (UniqueName: \"kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-token\") pod \"27621a94-aeca-461d-bf48-cddfb8c0ef53\" (UID: \"27621a94-aeca-461d-bf48-cddfb8c0ef53\") " Nov 03 20:29:23.896311 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.896260 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-socket" pod "27621a94-aeca-461d-bf48-cddfb8c0ef53" (UID: "27621a94-aeca-461d-bf48-cddfb8c0ef53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:23.896453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.896426 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27621a94-aeca-461d-bf48-cddfb8c0ef53-istiod-ca-cert" pod "27621a94-aeca-461d-bf48-cddfb8c0ef53" (UID: "27621a94-aeca-461d-bf48-cddfb8c0ef53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:23.897114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.897091 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.897114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.897115 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.897279 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.897125 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.897279 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.897134 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.897279 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.897142 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27621a94-aeca-461d-bf48-cddfb8c0ef53-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.898893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.898852 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-kube-api-access-q6jd9" pod "27621a94-aeca-461d-bf48-cddfb8c0ef53" (UID: "27621a94-aeca-461d-bf48-cddfb8c0ef53"). InnerVolumeSpecName "kube-api-access-q6jd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:23.898893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.898858 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-podinfo" pod "27621a94-aeca-461d-bf48-cddfb8c0ef53" (UID: "27621a94-aeca-461d-bf48-cddfb8c0ef53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:29:23.899504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.899480 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-token" pod "27621a94-aeca-461d-bf48-cddfb8c0ef53" (UID: "27621a94-aeca-461d-bf48-cddfb8c0ef53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:23.899605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.899481 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-envoy" pod "27621a94-aeca-461d-bf48-cddfb8c0ef53" (UID: "27621a94-aeca-461d-bf48-cddfb8c0ef53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:29:23.998670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.998578 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6jd9\" (UniqueName: \"kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-kube-api-access-q6jd9\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.998670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.998611 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.998670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.998627 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:23.998670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:23.998639 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27621a94-aeca-461d-bf48-cddfb8c0ef53-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:24.829250 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:24.829218 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw" Nov 03 20:29:24.857620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:24.857579 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw"] Nov 03 20:29:24.863243 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:24.863214 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-bfsxw"] Nov 03 20:29:25.765925 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:25.765887 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27621a94-aeca-461d-bf48-cddfb8c0ef53" path="/var/lib/kubelet/pods/27621a94-aeca-461d-bf48-cddfb8c0ef53/volumes" Nov 03 20:29:25.766894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:25.766866 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc705cd8-f6b5-4907-a90c-39f0c779f227" path="/var/lib/kubelet/pods/cc705cd8-f6b5-4907-a90c-39f0c779f227/volumes" Nov 03 20:29:28.033024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:28.032990 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f"] Nov 03 20:29:28.049999 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:28.049965 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7fd1a7-2mb7f"] Nov 03 20:29:29.764753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:29.764717 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1787b9bc-7db1-4195-b51e-fa5b705f3124" path="/var/lib/kubelet/pods/1787b9bc-7db1-4195-b51e-fa5b705f3124/volumes" Nov 03 20:29:36.004012 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.003974 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d"] Nov 03 20:29:36.005253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005230 4457 cpu_manager.go:414] "containerMap: removing container" podUID="cc705cd8-f6b5-4907-a90c-39f0c779f227" containerName="payload-pre-processing" Nov 03 20:29:36.005253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005251 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc705cd8-f6b5-4907-a90c-39f0c779f227" containerName="payload-pre-processing" Nov 03 20:29:36.005464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005263 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" Nov 03 20:29:36.005464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005270 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" Nov 03 20:29:36.005464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005309 4457 cpu_manager.go:414] "containerMap: removing container" podUID="1787b9bc-7db1-4195-b51e-fa5b705f3124" containerName="revoke-keys" Nov 03 20:29:36.005464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005314 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1787b9bc-7db1-4195-b51e-fa5b705f3124" containerName="revoke-keys" Nov 03 20:29:36.005464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005329 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" Nov 03 20:29:36.005464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005334 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" Nov 03 20:29:36.005464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005350 4457 cpu_manager.go:414] "containerMap: removing container" podUID="27621a94-aeca-461d-bf48-cddfb8c0ef53" containerName="istio-proxy" Nov 03 20:29:36.005464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005354 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27621a94-aeca-461d-bf48-cddfb8c0ef53" containerName="istio-proxy" Nov 03 20:29:36.005737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005530 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="1787b9bc-7db1-4195-b51e-fa5b705f3124" containerName="revoke-keys" Nov 03 20:29:36.005737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005540 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" Nov 03 20:29:36.005737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005547 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="27621a94-aeca-461d-bf48-cddfb8c0ef53" containerName="istio-proxy" Nov 03 20:29:36.005737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005557 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc705cd8-f6b5-4907-a90c-39f0c779f227" containerName="payload-pre-processing" Nov 03 20:29:36.005737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.005563 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0386779-b087-4c1b-9c9e-0a38d3e9a8d2" containerName="istio-proxy" Nov 03 20:29:36.011749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.011451 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" Nov 03 20:29:36.016637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.016613 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d"] Nov 03 20:29:36.021572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.021379 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:29:36.021652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.021607 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:29:36.151158 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:29:36.151118 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:29:36.165540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.165501 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d\" (UID: \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" Nov 03 20:29:36.165734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.165672 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqnr6\" (UniqueName: \"kubernetes.io/projected/a8ae20ec-001e-42e2-a480-45c6121fd0f7-kube-api-access-fqnr6\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d\" (UID: \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" Nov 03 20:29:36.165875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.165849 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d\" (UID: \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" Nov 03 20:29:36.267738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.267638 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d\" (UID: \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" Nov 03 20:29:36.267962 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.267782 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqnr6\" (UniqueName: \"kubernetes.io/projected/a8ae20ec-001e-42e2-a480-45c6121fd0f7-kube-api-access-fqnr6\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d\" (UID: \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" Nov 03 20:29:36.267962 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.267936 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d\" (UID: \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" Nov 03 20:29:36.269212 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.269158 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d\" (UID: \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" Nov 03 20:29:36.276405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.271464 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d\" (UID: \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" Nov 03 20:29:36.281060 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.281031 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqnr6\" (UniqueName: \"kubernetes.io/projected/a8ae20ec-001e-42e2-a480-45c6121fd0f7-kube-api-access-fqnr6\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d\" (UID: \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" Nov 03 20:29:36.329176 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.329142 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" Nov 03 20:29:36.632074 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:29:36.632044 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ae20ec_001e_42e2_a480_45c6121fd0f7.slice/crio-a70415505ab347be279014d4a5750ea2b733880a08c78ffb452af0207cd02d18 WatchSource:0}: Error finding container a70415505ab347be279014d4a5750ea2b733880a08c78ffb452af0207cd02d18: Status 404 returned error can't find the container with id a70415505ab347be279014d4a5750ea2b733880a08c78ffb452af0207cd02d18 Nov 03 20:29:36.633720 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:36.633693 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d"] Nov 03 20:29:37.130187 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:37.130154 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" event={"ID":"a8ae20ec-001e-42e2-a480-45c6121fd0f7","Type":"ContainerStarted","Data":"a70415505ab347be279014d4a5750ea2b733880a08c78ffb452af0207cd02d18"} Nov 03 20:29:38.168186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:38.168152 4457 generic.go:356] "Generic (PLEG): container finished" podID="a8ae20ec-001e-42e2-a480-45c6121fd0f7" containerID="575f3c1efe760867cdd55d210d287eafe57b007c2cada8e35fa2cc06b7ca6026" exitCode=0 Nov 03 20:29:38.168771 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:38.168244 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" event={"ID":"a8ae20ec-001e-42e2-a480-45c6121fd0f7","Type":"ContainerDied","Data":"575f3c1efe760867cdd55d210d287eafe57b007c2cada8e35fa2cc06b7ca6026"} Nov 03 20:29:39.376428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:39.376375 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" Nov 03 20:29:39.536543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:39.536463 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ae20ec-001e-42e2-a480-45c6121fd0f7-kube-api-access-fqnr6\" (UniqueName: \"kubernetes.io/projected/a8ae20ec-001e-42e2-a480-45c6121fd0f7-kube-api-access-fqnr6\") pod \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\" (UID: \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\") " Nov 03 20:29:39.536702 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:39.536665 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-openshift-ca\") pod \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\" (UID: \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\") " Nov 03 20:29:39.536702 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:39.536687 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-xks-ca\") pod \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\" (UID: \"a8ae20ec-001e-42e2-a480-45c6121fd0f7\") " Nov 03 20:29:39.537072 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:39.537039 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-openshift-ca" pod "a8ae20ec-001e-42e2-a480-45c6121fd0f7" (UID: "a8ae20ec-001e-42e2-a480-45c6121fd0f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:39.537454 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:39.537433 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:39.538664 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:39.538638 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ae20ec-001e-42e2-a480-45c6121fd0f7-kube-api-access-fqnr6" pod "a8ae20ec-001e-42e2-a480-45c6121fd0f7" (UID: "a8ae20ec-001e-42e2-a480-45c6121fd0f7"). InnerVolumeSpecName "kube-api-access-fqnr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:39.538797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:39.538783 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-xks-ca" pod "a8ae20ec-001e-42e2-a480-45c6121fd0f7" (UID: "a8ae20ec-001e-42e2-a480-45c6121fd0f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:29:39.638682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:39.638643 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8ae20ec-001e-42e2-a480-45c6121fd0f7-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:39.638682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:39.638676 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqnr6\" (UniqueName: \"kubernetes.io/projected/a8ae20ec-001e-42e2-a480-45c6121fd0f7-kube-api-access-fqnr6\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:40.194208 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:40.194176 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" Nov 03 20:29:40.194427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:40.194200 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d" event={"ID":"a8ae20ec-001e-42e2-a480-45c6121fd0f7","Type":"ContainerDied","Data":"a70415505ab347be279014d4a5750ea2b733880a08c78ffb452af0207cd02d18"} Nov 03 20:29:40.194427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:40.194257 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a70415505ab347be279014d4a5750ea2b733880a08c78ffb452af0207cd02d18" Nov 03 20:29:46.139095 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:46.139060 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc"] Nov 03 20:29:46.146305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:46.146276 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7"] Nov 03 20:29:51.140183 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.140132 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" podUID="81847f97-4b4e-460a-9a37-6ea2e76bb1d4" containerName="payload-processing" containerID="cri-o://1e88280b432bade611b5a1bd57645b5e06b426a3ba9e0e0e7030d22a11a62f24" gracePeriod=25 Nov 03 20:29:51.147361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.147290 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" podUID="0f09f12d-25df-412a-8090-101261a0d96e" containerName="payload-pre-processing" containerID="cri-o://6922c1f140ca25466434dd441b7139dad872aeaa61c8300cb33ddd883f840314" gracePeriod=25 Nov 03 20:29:51.349402 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.349179 4457 generic.go:356] "Generic (PLEG): container finished" podID="81847f97-4b4e-460a-9a37-6ea2e76bb1d4" containerID="1e88280b432bade611b5a1bd57645b5e06b426a3ba9e0e0e7030d22a11a62f24" exitCode=0 Nov 03 20:29:51.349402 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.349296 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" event={"ID":"81847f97-4b4e-460a-9a37-6ea2e76bb1d4","Type":"ContainerDied","Data":"1e88280b432bade611b5a1bd57645b5e06b426a3ba9e0e0e7030d22a11a62f24"} Nov 03 20:29:51.352091 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.352062 4457 generic.go:356] "Generic (PLEG): container finished" podID="0f09f12d-25df-412a-8090-101261a0d96e" containerID="6922c1f140ca25466434dd441b7139dad872aeaa61c8300cb33ddd883f840314" exitCode=0 Nov 03 20:29:51.352335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.352268 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" event={"ID":"0f09f12d-25df-412a-8090-101261a0d96e","Type":"ContainerDied","Data":"6922c1f140ca25466434dd441b7139dad872aeaa61c8300cb33ddd883f840314"} Nov 03 20:29:51.464589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.464548 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" Nov 03 20:29:51.530825 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.530789 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-plugins-config-volume\") pod \"81847f97-4b4e-460a-9a37-6ea2e76bb1d4\" (UID: \"81847f97-4b4e-460a-9a37-6ea2e76bb1d4\") " Nov 03 20:29:51.530999 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.530863 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-kube-api-access-46vrl\" (UniqueName: \"kubernetes.io/projected/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-kube-api-access-46vrl\") pod \"81847f97-4b4e-460a-9a37-6ea2e76bb1d4\" (UID: \"81847f97-4b4e-460a-9a37-6ea2e76bb1d4\") " Nov 03 20:29:51.531250 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.531225 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-plugins-config-volume" pod "81847f97-4b4e-460a-9a37-6ea2e76bb1d4" (UID: "81847f97-4b4e-460a-9a37-6ea2e76bb1d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:51.532130 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.531853 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:51.533574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.533549 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" Nov 03 20:29:51.534584 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.534562 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-kube-api-access-46vrl" pod "81847f97-4b4e-460a-9a37-6ea2e76bb1d4" (UID: "81847f97-4b4e-460a-9a37-6ea2e76bb1d4"). InnerVolumeSpecName "kube-api-access-46vrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:51.693568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.693082 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46vrl\" (UniqueName: \"kubernetes.io/projected/81847f97-4b4e-460a-9a37-6ea2e76bb1d4-kube-api-access-46vrl\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:51.801427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.799974 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f09f12d-25df-412a-8090-101261a0d96e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f09f12d-25df-412a-8090-101261a0d96e-plugins-config-volume\") pod \"0f09f12d-25df-412a-8090-101261a0d96e\" (UID: \"0f09f12d-25df-412a-8090-101261a0d96e\") " Nov 03 20:29:51.801427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.800249 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f09f12d-25df-412a-8090-101261a0d96e-kube-api-access-94p7l\" (UniqueName: \"kubernetes.io/projected/0f09f12d-25df-412a-8090-101261a0d96e-kube-api-access-94p7l\") pod \"0f09f12d-25df-412a-8090-101261a0d96e\" (UID: \"0f09f12d-25df-412a-8090-101261a0d96e\") " Nov 03 20:29:51.802301 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.801988 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f09f12d-25df-412a-8090-101261a0d96e-plugins-config-volume" pod "0f09f12d-25df-412a-8090-101261a0d96e" (UID: "0f09f12d-25df-412a-8090-101261a0d96e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:29:51.804679 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.804650 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f09f12d-25df-412a-8090-101261a0d96e-kube-api-access-94p7l" pod "0f09f12d-25df-412a-8090-101261a0d96e" (UID: "0f09f12d-25df-412a-8090-101261a0d96e"). InnerVolumeSpecName "kube-api-access-94p7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:29:51.902917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.902880 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94p7l\" (UniqueName: \"kubernetes.io/projected/0f09f12d-25df-412a-8090-101261a0d96e-kube-api-access-94p7l\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:51.903126 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:51.902931 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f09f12d-25df-412a-8090-101261a0d96e-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:29:52.365020 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:52.364991 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" Nov 03 20:29:52.365020 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:52.365003 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc" event={"ID":"81847f97-4b4e-460a-9a37-6ea2e76bb1d4","Type":"ContainerDied","Data":"40530a0c97dbc6dd7c7e6d73255dd331ad1e1427b8707fc81d0e7497d68cd094"} Nov 03 20:29:52.365651 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:52.365075 4457 scope.go:122] "RemoveContainer" containerID="1e88280b432bade611b5a1bd57645b5e06b426a3ba9e0e0e7030d22a11a62f24" Nov 03 20:29:52.367061 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:52.367037 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" Nov 03 20:29:52.367329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:52.367308 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7" event={"ID":"0f09f12d-25df-412a-8090-101261a0d96e","Type":"ContainerDied","Data":"5fdd29c8bb38631fe2a0e49293aa18ffde7a8b5f615562858d07e8ff99cc3b08"} Nov 03 20:29:52.378646 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:52.378621 4457 scope.go:122] "RemoveContainer" containerID="6922c1f140ca25466434dd441b7139dad872aeaa61c8300cb33ddd883f840314" Nov 03 20:29:52.396673 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:52.396646 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc"] Nov 03 20:29:52.404941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:52.404914 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-683ba6-55ffc9c5fb-sl5kc"] Nov 03 20:29:52.413329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:52.413305 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7"] Nov 03 20:29:52.418360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:52.418338 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-683ba6-7cf6874c8c-d9dm7"] Nov 03 20:29:53.766792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:53.766746 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f09f12d-25df-412a-8090-101261a0d96e" path="/var/lib/kubelet/pods/0f09f12d-25df-412a-8090-101261a0d96e/volumes" Nov 03 20:29:53.767482 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:29:53.767459 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81847f97-4b4e-460a-9a37-6ea2e76bb1d4" path="/var/lib/kubelet/pods/81847f97-4b4e-460a-9a37-6ea2e76bb1d4/volumes" Nov 03 20:30:00.161916 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.161881 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm"] Nov 03 20:30:00.185945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.185114 4457 cpu_manager.go:414] "containerMap: removing container" podUID="81847f97-4b4e-460a-9a37-6ea2e76bb1d4" containerName="payload-processing" Nov 03 20:30:00.186613 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.186543 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81847f97-4b4e-460a-9a37-6ea2e76bb1d4" containerName="payload-processing" Nov 03 20:30:00.186801 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.186786 4457 cpu_manager.go:414] "containerMap: removing container" podUID="a8ae20ec-001e-42e2-a480-45c6121fd0f7" containerName="revoke-keys" Nov 03 20:30:00.187202 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.186883 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ae20ec-001e-42e2-a480-45c6121fd0f7" containerName="revoke-keys" Nov 03 20:30:00.187202 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.187002 4457 cpu_manager.go:414] "containerMap: removing container" podUID="0f09f12d-25df-412a-8090-101261a0d96e" containerName="payload-pre-processing" Nov 03 20:30:00.187202 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.187011 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f09f12d-25df-412a-8090-101261a0d96e" containerName="payload-pre-processing" Nov 03 20:30:00.188517 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.188486 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="81847f97-4b4e-460a-9a37-6ea2e76bb1d4" containerName="payload-processing" Nov 03 20:30:00.188826 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.188811 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ae20ec-001e-42e2-a480-45c6121fd0f7" containerName="revoke-keys" Nov 03 20:30:00.188989 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.188974 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f09f12d-25df-412a-8090-101261a0d96e" containerName="payload-pre-processing" Nov 03 20:30:00.195495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.193944 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" Nov 03 20:30:00.213047 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.211668 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr"] Nov 03 20:30:00.218915 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.218328 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" Nov 03 20:30:00.231409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.227472 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5"] Nov 03 20:30:00.239493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.239331 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" Nov 03 20:30:00.257882 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.257845 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp"] Nov 03 20:30:00.321068 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.304038 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:30:00.325113 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.322933 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9fce91d-067e-413a-8d5b-85d5b3d04e80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm\" (UID: \"c9fce91d-067e-413a-8d5b-85d5b3d04e80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" Nov 03 20:30:00.325113 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.322995 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxc8n\" (UniqueName: \"kubernetes.io/projected/c9fce91d-067e-413a-8d5b-85d5b3d04e80-kube-api-access-sxc8n\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm\" (UID: \"c9fce91d-067e-413a-8d5b-85d5b3d04e80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" Nov 03 20:30:00.325113 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.323181 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s4mv\" (UniqueName: \"kubernetes.io/projected/f52a0620-190b-418a-afdd-8c8ac053a1d4-kube-api-access-4s4mv\") pod \"maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr\" (UID: \"f52a0620-190b-418a-afdd-8c8ac053a1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" Nov 03 20:30:00.325113 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.323622 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bx6r\" (UniqueName: \"kubernetes.io/projected/ce8d2d9a-946c-4e76-b319-c7d653ac2306-kube-api-access-4bx6r\") pod \"maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5\" (UID: \"ce8d2d9a-946c-4e76-b319-c7d653ac2306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" Nov 03 20:30:00.325113 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.323727 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52a0620-190b-418a-afdd-8c8ac053a1d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr\" (UID: \"f52a0620-190b-418a-afdd-8c8ac053a1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" Nov 03 20:30:00.325113 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.323767 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce8d2d9a-946c-4e76-b319-c7d653ac2306-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5\" (UID: \"ce8d2d9a-946c-4e76-b319-c7d653ac2306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" Nov 03 20:30:00.364443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.364375 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc"] Nov 03 20:30:00.367854 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.365170 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" Nov 03 20:30:00.382632 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.381331 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" Nov 03 20:30:00.422184 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.422103 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7"] Nov 03 20:30:00.425780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.425743 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9fce91d-067e-413a-8d5b-85d5b3d04e80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm\" (UID: \"c9fce91d-067e-413a-8d5b-85d5b3d04e80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" Nov 03 20:30:00.426517 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.426491 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9fce91d-067e-413a-8d5b-85d5b3d04e80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm\" (UID: \"c9fce91d-067e-413a-8d5b-85d5b3d04e80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" Nov 03 20:30:00.428000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.426592 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxc8n\" (UniqueName: \"kubernetes.io/projected/c9fce91d-067e-413a-8d5b-85d5b3d04e80-kube-api-access-sxc8n\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm\" (UID: \"c9fce91d-067e-413a-8d5b-85d5b3d04e80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" Nov 03 20:30:00.428166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.428148 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s4mv\" (UniqueName: \"kubernetes.io/projected/f52a0620-190b-418a-afdd-8c8ac053a1d4-kube-api-access-4s4mv\") pod \"maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr\" (UID: \"f52a0620-190b-418a-afdd-8c8ac053a1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" Nov 03 20:30:00.428346 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.428330 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bx6r\" (UniqueName: \"kubernetes.io/projected/ce8d2d9a-946c-4e76-b319-c7d653ac2306-kube-api-access-4bx6r\") pod \"maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5\" (UID: \"ce8d2d9a-946c-4e76-b319-c7d653ac2306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" Nov 03 20:30:00.428453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.428437 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52a0620-190b-418a-afdd-8c8ac053a1d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr\" (UID: \"f52a0620-190b-418a-afdd-8c8ac053a1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" Nov 03 20:30:00.428520 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.428479 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce8d2d9a-946c-4e76-b319-c7d653ac2306-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5\" (UID: \"ce8d2d9a-946c-4e76-b319-c7d653ac2306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" Nov 03 20:30:00.429708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.429684 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52a0620-190b-418a-afdd-8c8ac053a1d4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr\" (UID: \"f52a0620-190b-418a-afdd-8c8ac053a1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" Nov 03 20:30:00.430082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.430067 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" Nov 03 20:30:00.430354 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.430333 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce8d2d9a-946c-4e76-b319-c7d653ac2306-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5\" (UID: \"ce8d2d9a-946c-4e76-b319-c7d653ac2306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" Nov 03 20:30:00.433317 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.433292 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr"] Nov 03 20:30:00.440538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.440503 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm"] Nov 03 20:30:00.448417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.448377 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5"] Nov 03 20:30:00.457407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.457366 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7"] Nov 03 20:30:00.466678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.466652 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp"] Nov 03 20:30:00.473148 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.473126 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" Nov 03 20:30:00.481831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.481802 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp"] Nov 03 20:30:00.513405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.500464 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc"] Nov 03 20:30:00.533758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.533723 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rcjn\" (UniqueName: \"kubernetes.io/projected/ff06a2f7-0660-4ffb-83c8-f362e3e89658-kube-api-access-2rcjn\") pod \"maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc\" (UID: \"ff06a2f7-0660-4ffb-83c8-f362e3e89658\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" Nov 03 20:30:00.533940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.533823 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrx8d\" (UniqueName: \"kubernetes.io/projected/541df3aa-7e72-4bfc-a895-b4c04ea1247e-kube-api-access-qrx8d\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp\" (UID: \"541df3aa-7e72-4bfc-a895-b4c04ea1247e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" Nov 03 20:30:00.533940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.533918 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541df3aa-7e72-4bfc-a895-b4c04ea1247e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp\" (UID: \"541df3aa-7e72-4bfc-a895-b4c04ea1247e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" Nov 03 20:30:00.534117 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.533946 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff06a2f7-0660-4ffb-83c8-f362e3e89658-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc\" (UID: \"ff06a2f7-0660-4ffb-83c8-f362e3e89658\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" Nov 03 20:30:00.534117 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.534036 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9wms\" (UniqueName: \"kubernetes.io/projected/e68e63cf-3ae5-4035-b096-b8d0facc69c0-kube-api-access-c9wms\") pod \"maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7\" (UID: \"e68e63cf-3ae5-4035-b096-b8d0facc69c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" Nov 03 20:30:00.534374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.534357 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68e63cf-3ae5-4035-b096-b8d0facc69c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7\" (UID: \"e68e63cf-3ae5-4035-b096-b8d0facc69c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" Nov 03 20:30:00.535585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.535548 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxc8n\" (UniqueName: \"kubernetes.io/projected/c9fce91d-067e-413a-8d5b-85d5b3d04e80-kube-api-access-sxc8n\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm\" (UID: \"c9fce91d-067e-413a-8d5b-85d5b3d04e80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" Nov 03 20:30:00.538438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.538412 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp"] Nov 03 20:30:00.539277 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.539251 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bx6r\" (UniqueName: \"kubernetes.io/projected/ce8d2d9a-946c-4e76-b319-c7d653ac2306-kube-api-access-4bx6r\") pod \"maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5\" (UID: \"ce8d2d9a-946c-4e76-b319-c7d653ac2306\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" Nov 03 20:30:00.540152 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.539267 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s4mv\" (UniqueName: \"kubernetes.io/projected/f52a0620-190b-418a-afdd-8c8ac053a1d4-kube-api-access-4s4mv\") pod \"maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr\" (UID: \"f52a0620-190b-418a-afdd-8c8ac053a1d4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" Nov 03 20:30:00.549997 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.549973 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv"] Nov 03 20:30:00.555595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.555572 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" Nov 03 20:30:00.562253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.562225 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n"] Nov 03 20:30:00.571462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.571440 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" Nov 03 20:30:00.576694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.576670 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" Nov 03 20:30:00.609471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.597051 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv"] Nov 03 20:30:00.609471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.606858 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n"] Nov 03 20:30:00.641640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.638434 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zbpg\" (UniqueName: \"kubernetes.io/projected/d168551a-ced0-4313-83f4-6f1d4f3b758d-kube-api-access-7zbpg\") pod \"maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp\" (UID: \"d168551a-ced0-4313-83f4-6f1d4f3b758d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" Nov 03 20:30:00.641640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.638604 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92n2d\" (UniqueName: \"kubernetes.io/projected/6cad965b-7c57-4560-921e-0e5ed244784a-kube-api-access-92n2d\") pod \"maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv\" (UID: \"6cad965b-7c57-4560-921e-0e5ed244784a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" Nov 03 20:30:00.641640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.638765 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68e63cf-3ae5-4035-b096-b8d0facc69c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7\" (UID: \"e68e63cf-3ae5-4035-b096-b8d0facc69c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" Nov 03 20:30:00.641640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.638845 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rcjn\" (UniqueName: \"kubernetes.io/projected/ff06a2f7-0660-4ffb-83c8-f362e3e89658-kube-api-access-2rcjn\") pod \"maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc\" (UID: \"ff06a2f7-0660-4ffb-83c8-f362e3e89658\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" Nov 03 20:30:00.641640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.638916 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrx8d\" (UniqueName: \"kubernetes.io/projected/541df3aa-7e72-4bfc-a895-b4c04ea1247e-kube-api-access-qrx8d\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp\" (UID: \"541df3aa-7e72-4bfc-a895-b4c04ea1247e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" Nov 03 20:30:00.641640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.638957 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d168551a-ced0-4313-83f4-6f1d4f3b758d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp\" (UID: \"d168551a-ced0-4313-83f4-6f1d4f3b758d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" Nov 03 20:30:00.641640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.638999 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cad965b-7c57-4560-921e-0e5ed244784a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv\" (UID: \"6cad965b-7c57-4560-921e-0e5ed244784a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" Nov 03 20:30:00.641640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.639030 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541df3aa-7e72-4bfc-a895-b4c04ea1247e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp\" (UID: \"541df3aa-7e72-4bfc-a895-b4c04ea1247e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" Nov 03 20:30:00.641640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.639061 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff06a2f7-0660-4ffb-83c8-f362e3e89658-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc\" (UID: \"ff06a2f7-0660-4ffb-83c8-f362e3e89658\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" Nov 03 20:30:00.641640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.639150 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9wms\" (UniqueName: \"kubernetes.io/projected/e68e63cf-3ae5-4035-b096-b8d0facc69c0-kube-api-access-c9wms\") pod \"maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7\" (UID: \"e68e63cf-3ae5-4035-b096-b8d0facc69c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" Nov 03 20:30:00.646880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.646782 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541df3aa-7e72-4bfc-a895-b4c04ea1247e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp\" (UID: \"541df3aa-7e72-4bfc-a895-b4c04ea1247e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" Nov 03 20:30:00.647853 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.647821 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff06a2f7-0660-4ffb-83c8-f362e3e89658-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc\" (UID: \"ff06a2f7-0660-4ffb-83c8-f362e3e89658\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" Nov 03 20:30:00.647977 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.628354 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d"] Nov 03 20:30:00.652892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.652714 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" Nov 03 20:30:00.652892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.652844 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68e63cf-3ae5-4035-b096-b8d0facc69c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7\" (UID: \"e68e63cf-3ae5-4035-b096-b8d0facc69c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" Nov 03 20:30:00.709615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.682489 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrx8d\" (UniqueName: \"kubernetes.io/projected/541df3aa-7e72-4bfc-a895-b4c04ea1247e-kube-api-access-qrx8d\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp\" (UID: \"541df3aa-7e72-4bfc-a895-b4c04ea1247e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" Nov 03 20:30:00.709615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.683162 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9wms\" (UniqueName: \"kubernetes.io/projected/e68e63cf-3ae5-4035-b096-b8d0facc69c0-kube-api-access-c9wms\") pod \"maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7\" (UID: \"e68e63cf-3ae5-4035-b096-b8d0facc69c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" Nov 03 20:30:00.709615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.685509 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" Nov 03 20:30:00.709615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.685772 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rcjn\" (UniqueName: \"kubernetes.io/projected/ff06a2f7-0660-4ffb-83c8-f362e3e89658-kube-api-access-2rcjn\") pod \"maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc\" (UID: \"ff06a2f7-0660-4ffb-83c8-f362e3e89658\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" Nov 03 20:30:00.709615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.697233 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k"] Nov 03 20:30:00.718671 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.718638 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-683ba6-j599d"] Nov 03 20:30:00.718827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.718689 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k"] Nov 03 20:30:00.718827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.718723 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" Nov 03 20:30:00.726151 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.726120 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" Nov 03 20:30:00.730557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.730536 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl"] Nov 03 20:30:00.739448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.738629 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" Nov 03 20:30:00.741461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.741434 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" Nov 03 20:30:00.744535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.743900 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clfds\" (UniqueName: \"kubernetes.io/projected/b50409d7-ddee-4335-86eb-34b5b0f87924-kube-api-access-clfds\") pod \"maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n\" (UID: \"b50409d7-ddee-4335-86eb-34b5b0f87924\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" Nov 03 20:30:00.744535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.744063 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d168551a-ced0-4313-83f4-6f1d4f3b758d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp\" (UID: \"d168551a-ced0-4313-83f4-6f1d4f3b758d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" Nov 03 20:30:00.745476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.744950 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d168551a-ced0-4313-83f4-6f1d4f3b758d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp\" (UID: \"d168551a-ced0-4313-83f4-6f1d4f3b758d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" Nov 03 20:30:00.745476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.744112 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cad965b-7c57-4560-921e-0e5ed244784a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv\" (UID: \"6cad965b-7c57-4560-921e-0e5ed244784a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" Nov 03 20:30:00.745476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.745457 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zbpg\" (UniqueName: \"kubernetes.io/projected/d168551a-ced0-4313-83f4-6f1d4f3b758d-kube-api-access-7zbpg\") pod \"maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp\" (UID: \"d168551a-ced0-4313-83f4-6f1d4f3b758d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" Nov 03 20:30:00.745766 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.745378 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cad965b-7c57-4560-921e-0e5ed244784a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv\" (UID: \"6cad965b-7c57-4560-921e-0e5ed244784a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" Nov 03 20:30:00.746584 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.746557 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50409d7-ddee-4335-86eb-34b5b0f87924-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n\" (UID: \"b50409d7-ddee-4335-86eb-34b5b0f87924\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" Nov 03 20:30:00.747344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.747287 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92n2d\" (UniqueName: \"kubernetes.io/projected/6cad965b-7c57-4560-921e-0e5ed244784a-kube-api-access-92n2d\") pod \"maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv\" (UID: \"6cad965b-7c57-4560-921e-0e5ed244784a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" Nov 03 20:30:00.750349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.750325 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl"] Nov 03 20:30:00.775420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.773663 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" Nov 03 20:30:00.850660 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.850380 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50409d7-ddee-4335-86eb-34b5b0f87924-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n\" (UID: \"b50409d7-ddee-4335-86eb-34b5b0f87924\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" Nov 03 20:30:00.850660 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.850545 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k\" (UID: \"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" Nov 03 20:30:00.851428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.850867 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl\" (UID: \"eea2c970-9b94-45c7-b8e7-e1263a11f8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" Nov 03 20:30:00.851428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.850898 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clfds\" (UniqueName: \"kubernetes.io/projected/b50409d7-ddee-4335-86eb-34b5b0f87924-kube-api-access-clfds\") pod \"maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n\" (UID: \"b50409d7-ddee-4335-86eb-34b5b0f87924\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" Nov 03 20:30:00.851428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.850951 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jgpf\" (UniqueName: \"kubernetes.io/projected/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-kube-api-access-9jgpf\") pod \"maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl\" (UID: \"eea2c970-9b94-45c7-b8e7-e1263a11f8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" Nov 03 20:30:00.851428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.851012 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57hcr\" (UniqueName: \"kubernetes.io/projected/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-kube-api-access-57hcr\") pod \"maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k\" (UID: \"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" Nov 03 20:30:00.853126 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.852582 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50409d7-ddee-4335-86eb-34b5b0f87924-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n\" (UID: \"b50409d7-ddee-4335-86eb-34b5b0f87924\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" Nov 03 20:30:00.884720 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.884628 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zbpg\" (UniqueName: \"kubernetes.io/projected/d168551a-ced0-4313-83f4-6f1d4f3b758d-kube-api-access-7zbpg\") pod \"maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp\" (UID: \"d168551a-ced0-4313-83f4-6f1d4f3b758d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" Nov 03 20:30:00.892544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.888543 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92n2d\" (UniqueName: \"kubernetes.io/projected/6cad965b-7c57-4560-921e-0e5ed244784a-kube-api-access-92n2d\") pod \"maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv\" (UID: \"6cad965b-7c57-4560-921e-0e5ed244784a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" Nov 03 20:30:00.930024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.926844 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clfds\" (UniqueName: \"kubernetes.io/projected/b50409d7-ddee-4335-86eb-34b5b0f87924-kube-api-access-clfds\") pod \"maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n\" (UID: \"b50409d7-ddee-4335-86eb-34b5b0f87924\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" Nov 03 20:30:00.956184 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.956146 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k\" (UID: \"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" Nov 03 20:30:00.956554 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.956520 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl\" (UID: \"eea2c970-9b94-45c7-b8e7-e1263a11f8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" Nov 03 20:30:00.956769 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.956748 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jgpf\" (UniqueName: \"kubernetes.io/projected/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-kube-api-access-9jgpf\") pod \"maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl\" (UID: \"eea2c970-9b94-45c7-b8e7-e1263a11f8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" Nov 03 20:30:00.956978 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.956960 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57hcr\" (UniqueName: \"kubernetes.io/projected/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-kube-api-access-57hcr\") pod \"maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k\" (UID: \"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" Nov 03 20:30:00.960630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.960511 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k\" (UID: \"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" Nov 03 20:30:00.962021 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.962001 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl\" (UID: \"eea2c970-9b94-45c7-b8e7-e1263a11f8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" Nov 03 20:30:00.970544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.970443 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57hcr\" (UniqueName: \"kubernetes.io/projected/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-kube-api-access-57hcr\") pod \"maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k\" (UID: \"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" Nov 03 20:30:00.973546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:00.973176 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jgpf\" (UniqueName: \"kubernetes.io/projected/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-kube-api-access-9jgpf\") pod \"maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl\" (UID: \"eea2c970-9b94-45c7-b8e7-e1263a11f8f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" Nov 03 20:30:01.044180 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.043968 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" Nov 03 20:30:01.083921 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.083856 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" Nov 03 20:30:01.124824 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.124797 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" Nov 03 20:30:01.191203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.190957 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" Nov 03 20:30:01.205928 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.205706 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" Nov 03 20:30:01.218369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.218337 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm"] Nov 03 20:30:01.389470 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.389428 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc"] Nov 03 20:30:01.436378 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.436351 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp"] Nov 03 20:30:01.475312 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:30:01.472509 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541df3aa_7e72_4bfc_a895_b4c04ea1247e.slice/crio-9b36393683d2df7b73022320307271c5578f583207c3b0f8259b96fe75ff7862 WatchSource:0}: Error finding container 9b36393683d2df7b73022320307271c5578f583207c3b0f8259b96fe75ff7862: Status 404 returned error can't find the container with id 9b36393683d2df7b73022320307271c5578f583207c3b0f8259b96fe75ff7862 Nov 03 20:30:01.487107 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.487073 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" event={"ID":"c9fce91d-067e-413a-8d5b-85d5b3d04e80","Type":"ContainerStarted","Data":"ba851d17147bb729c36bc6dc92820dd4df4f63f9aaaf566faf8fc9606561f176"} Nov 03 20:30:01.489197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.489165 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" event={"ID":"ff06a2f7-0660-4ffb-83c8-f362e3e89658","Type":"ContainerStarted","Data":"1877d3ff6e2b19c2b7066297d650bbaad270b9e6a1be7b13ce00a9f21a9bad38"} Nov 03 20:30:01.516720 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.516688 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5"] Nov 03 20:30:01.555211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.555182 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr"] Nov 03 20:30:01.559004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.558976 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv"] Nov 03 20:30:01.628680 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.628647 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n"] Nov 03 20:30:01.640149 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:30:01.639732 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50409d7_ddee_4335_86eb_34b5b0f87924.slice/crio-b19f8bca27785626dbd4c90e9af761e0767281863902887147553766a747fa3b WatchSource:0}: Error finding container b19f8bca27785626dbd4c90e9af761e0767281863902887147553766a747fa3b: Status 404 returned error can't find the container with id b19f8bca27785626dbd4c90e9af761e0767281863902887147553766a747fa3b Nov 03 20:30:01.653315 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.653231 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7"] Nov 03 20:30:01.723743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.723714 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp"] Nov 03 20:30:01.809223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.809192 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ae20ec-001e-42e2-a480-45c6121fd0f7" path="/var/lib/kubelet/pods/a8ae20ec-001e-42e2-a480-45c6121fd0f7/volumes" Nov 03 20:30:01.813689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.813662 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k"] Nov 03 20:30:01.819002 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:01.818977 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl"] Nov 03 20:30:02.503121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.503087 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" event={"ID":"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121","Type":"ContainerStarted","Data":"8e8149882bf34bb56a9e10c57434288b8b1a03dc87e6d8e662a5430c9bea6a20"} Nov 03 20:30:02.505043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.504987 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" event={"ID":"b50409d7-ddee-4335-86eb-34b5b0f87924","Type":"ContainerStarted","Data":"b19f8bca27785626dbd4c90e9af761e0767281863902887147553766a747fa3b"} Nov 03 20:30:02.506557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.506534 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" event={"ID":"ce8d2d9a-946c-4e76-b319-c7d653ac2306","Type":"ContainerStarted","Data":"23da020ad82182ecd2f825912a4551e082e1190fe8f81cd8a78f5544db78a7af"} Nov 03 20:30:02.508123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.508100 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" event={"ID":"e68e63cf-3ae5-4035-b096-b8d0facc69c0","Type":"ContainerStarted","Data":"a15b999dfe926701399dc4bf5ddc5abf548d2cd7a3a2963ce1f80491c6b92646"} Nov 03 20:30:02.509694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.509674 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" event={"ID":"d168551a-ced0-4313-83f4-6f1d4f3b758d","Type":"ContainerStarted","Data":"0ed53dc20b47cd0e586916a61857a903f4b75402647c76f1500f5436ebc142df"} Nov 03 20:30:02.511689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.511659 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" event={"ID":"eea2c970-9b94-45c7-b8e7-e1263a11f8f1","Type":"ContainerStarted","Data":"a7e322a70c94ae145d2519f69c5786d4bc92a85d57e2d7035faa133419143733"} Nov 03 20:30:02.512878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.512854 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" event={"ID":"6cad965b-7c57-4560-921e-0e5ed244784a","Type":"ContainerStarted","Data":"312c4a9994b8480c99ce8fc4529252e3518a48751d9011dc6b7784cb643fe8b0"} Nov 03 20:30:02.514531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.514500 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" event={"ID":"c9fce91d-067e-413a-8d5b-85d5b3d04e80","Type":"ContainerStarted","Data":"a3469e4dbeb0947f4346edd0efb20510f044a87a1b90170dbb96ee053b7f3760"} Nov 03 20:30:02.543360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.543277 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" event={"ID":"f52a0620-190b-418a-afdd-8c8ac053a1d4","Type":"ContainerStarted","Data":"a23ceccad786e9ba08ebb988bb6902adadb622414b18412ce376f1b651cc86b6"} Nov 03 20:30:02.543360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.543350 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" event={"ID":"f52a0620-190b-418a-afdd-8c8ac053a1d4","Type":"ContainerStarted","Data":"d546ddc3e41d3b1d3dc1841df1d7257d3f5c6cb3f1df48646d77fe74599930ce"} Nov 03 20:30:02.549741 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.549685 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" podStartSLOduration=2.549669304 podStartE2EDuration="2.549669304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.545770041 +0000 UTC m=+1980.921366784" watchObservedRunningTime="2026-11-03 20:30:02.549669304 +0000 UTC m=+1980.925266054" Nov 03 20:30:02.550521 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.550474 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" event={"ID":"541df3aa-7e72-4bfc-a895-b4c04ea1247e","Type":"ContainerStarted","Data":"9b36393683d2df7b73022320307271c5578f583207c3b0f8259b96fe75ff7862"} Nov 03 20:30:02.579511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.564637 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" event={"ID":"ff06a2f7-0660-4ffb-83c8-f362e3e89658","Type":"ContainerStarted","Data":"2d97d019635f9773c674193bc6b908b89477fd6fc13eec0f99869e017a9b21db"} Nov 03 20:30:02.645155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.644523 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" podStartSLOduration=2.644497044 podStartE2EDuration="2.644497044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.63278386 +0000 UTC m=+1981.008380608" watchObservedRunningTime="2026-11-03 20:30:02.644497044 +0000 UTC m=+1981.020093791" Nov 03 20:30:02.659188 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:02.659133 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" podStartSLOduration=2.659113734 podStartE2EDuration="2.659113734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:02.655455522 +0000 UTC m=+1981.031052270" watchObservedRunningTime="2026-11-03 20:30:02.659113734 +0000 UTC m=+1981.034710482" Nov 03 20:30:03.581711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.581674 4457 generic.go:356] "Generic (PLEG): container finished" podID="d168551a-ced0-4313-83f4-6f1d4f3b758d" containerID="1169d6f533496a561bbdd7c98a1776a3daebdeeb323ecb4676bd5c010a6d2103" exitCode=60 Nov 03 20:30:03.582139 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.582041 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" event={"ID":"d168551a-ced0-4313-83f4-6f1d4f3b758d","Type":"ContainerDied","Data":"1169d6f533496a561bbdd7c98a1776a3daebdeeb323ecb4676bd5c010a6d2103"} Nov 03 20:30:03.582345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.582326 4457 scope.go:122] "RemoveContainer" containerID="1169d6f533496a561bbdd7c98a1776a3daebdeeb323ecb4676bd5c010a6d2103" Nov 03 20:30:03.584304 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.584275 4457 generic.go:356] "Generic (PLEG): container finished" podID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" containerID="c62893417b2af1014b7ecc51894ce8132ddc7f2a3103d58a321891252ad702f1" exitCode=60 Nov 03 20:30:03.584433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.584363 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" event={"ID":"eea2c970-9b94-45c7-b8e7-e1263a11f8f1","Type":"ContainerDied","Data":"c62893417b2af1014b7ecc51894ce8132ddc7f2a3103d58a321891252ad702f1"} Nov 03 20:30:03.585059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.584827 4457 scope.go:122] "RemoveContainer" containerID="c62893417b2af1014b7ecc51894ce8132ddc7f2a3103d58a321891252ad702f1" Nov 03 20:30:03.588674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.588642 4457 generic.go:356] "Generic (PLEG): container finished" podID="6cad965b-7c57-4560-921e-0e5ed244784a" containerID="7bed30139225ebcadf157f3529bb0e94dc991fbbe7a353e531ccdb87ba02549d" exitCode=60 Nov 03 20:30:03.588791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.588767 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" event={"ID":"6cad965b-7c57-4560-921e-0e5ed244784a","Type":"ContainerDied","Data":"7bed30139225ebcadf157f3529bb0e94dc991fbbe7a353e531ccdb87ba02549d"} Nov 03 20:30:03.589166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.589146 4457 scope.go:122] "RemoveContainer" containerID="7bed30139225ebcadf157f3529bb0e94dc991fbbe7a353e531ccdb87ba02549d" Nov 03 20:30:03.590870 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.590850 4457 generic.go:356] "Generic (PLEG): container finished" podID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" containerID="a3469e4dbeb0947f4346edd0efb20510f044a87a1b90170dbb96ee053b7f3760" exitCode=60 Nov 03 20:30:03.590951 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.590923 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" event={"ID":"c9fce91d-067e-413a-8d5b-85d5b3d04e80","Type":"ContainerDied","Data":"a3469e4dbeb0947f4346edd0efb20510f044a87a1b90170dbb96ee053b7f3760"} Nov 03 20:30:03.591278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.591263 4457 scope.go:122] "RemoveContainer" containerID="a3469e4dbeb0947f4346edd0efb20510f044a87a1b90170dbb96ee053b7f3760" Nov 03 20:30:03.594193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.594174 4457 generic.go:356] "Generic (PLEG): container finished" podID="f52a0620-190b-418a-afdd-8c8ac053a1d4" containerID="a23ceccad786e9ba08ebb988bb6902adadb622414b18412ce376f1b651cc86b6" exitCode=60 Nov 03 20:30:03.594287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.594260 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" event={"ID":"f52a0620-190b-418a-afdd-8c8ac053a1d4","Type":"ContainerDied","Data":"a23ceccad786e9ba08ebb988bb6902adadb622414b18412ce376f1b651cc86b6"} Nov 03 20:30:03.594504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.594484 4457 scope.go:122] "RemoveContainer" containerID="a23ceccad786e9ba08ebb988bb6902adadb622414b18412ce376f1b651cc86b6" Nov 03 20:30:03.609610 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.609582 4457 generic.go:356] "Generic (PLEG): container finished" podID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" containerID="9e54fa2154a7d83dda87e6192087485dffa0783c735f1d52300971213165e7e3" exitCode=60 Nov 03 20:30:03.609704 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.609690 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" event={"ID":"541df3aa-7e72-4bfc-a895-b4c04ea1247e","Type":"ContainerDied","Data":"9e54fa2154a7d83dda87e6192087485dffa0783c735f1d52300971213165e7e3"} Nov 03 20:30:03.609976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.609953 4457 scope.go:122] "RemoveContainer" containerID="9e54fa2154a7d83dda87e6192087485dffa0783c735f1d52300971213165e7e3" Nov 03 20:30:03.613052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.613025 4457 generic.go:356] "Generic (PLEG): container finished" podID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" containerID="2d97d019635f9773c674193bc6b908b89477fd6fc13eec0f99869e017a9b21db" exitCode=60 Nov 03 20:30:03.613182 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.613161 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" event={"ID":"ff06a2f7-0660-4ffb-83c8-f362e3e89658","Type":"ContainerDied","Data":"2d97d019635f9773c674193bc6b908b89477fd6fc13eec0f99869e017a9b21db"} Nov 03 20:30:03.613459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.613369 4457 scope.go:122] "RemoveContainer" containerID="2d97d019635f9773c674193bc6b908b89477fd6fc13eec0f99869e017a9b21db" Nov 03 20:30:03.615140 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.615118 4457 generic.go:356] "Generic (PLEG): container finished" podID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" containerID="430f8b7ebd97d9a8e4056be8e6d7aed19b479c585aebda4ecb01a085dc91fdbe" exitCode=60 Nov 03 20:30:03.615425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.615181 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" event={"ID":"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121","Type":"ContainerDied","Data":"430f8b7ebd97d9a8e4056be8e6d7aed19b479c585aebda4ecb01a085dc91fdbe"} Nov 03 20:30:03.616103 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.615509 4457 scope.go:122] "RemoveContainer" containerID="430f8b7ebd97d9a8e4056be8e6d7aed19b479c585aebda4ecb01a085dc91fdbe" Nov 03 20:30:03.617061 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.617041 4457 generic.go:356] "Generic (PLEG): container finished" podID="b50409d7-ddee-4335-86eb-34b5b0f87924" containerID="b2a4aa199dc2567526084c39ffb80ef8a7044d7e832ec1f89ebc451ba54f0277" exitCode=60 Nov 03 20:30:03.617147 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.617121 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" event={"ID":"b50409d7-ddee-4335-86eb-34b5b0f87924","Type":"ContainerDied","Data":"b2a4aa199dc2567526084c39ffb80ef8a7044d7e832ec1f89ebc451ba54f0277"} Nov 03 20:30:03.617376 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.617359 4457 scope.go:122] "RemoveContainer" containerID="b2a4aa199dc2567526084c39ffb80ef8a7044d7e832ec1f89ebc451ba54f0277" Nov 03 20:30:03.618787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.618766 4457 generic.go:356] "Generic (PLEG): container finished" podID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" containerID="0490ff09607bb50a0a95119004a3b468a081bd6f8c82c55b0f5b4a720a0c5408" exitCode=60 Nov 03 20:30:03.618864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.618838 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" event={"ID":"ce8d2d9a-946c-4e76-b319-c7d653ac2306","Type":"ContainerDied","Data":"0490ff09607bb50a0a95119004a3b468a081bd6f8c82c55b0f5b4a720a0c5408"} Nov 03 20:30:03.619042 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.619024 4457 scope.go:122] "RemoveContainer" containerID="0490ff09607bb50a0a95119004a3b468a081bd6f8c82c55b0f5b4a720a0c5408" Nov 03 20:30:03.621230 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.621206 4457 generic.go:356] "Generic (PLEG): container finished" podID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" containerID="0ce306148a68de58faad9127327925c6bc1ff93230225ca03c945e4037788ade" exitCode=60 Nov 03 20:30:03.621312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.621236 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" event={"ID":"e68e63cf-3ae5-4035-b096-b8d0facc69c0","Type":"ContainerDied","Data":"0ce306148a68de58faad9127327925c6bc1ff93230225ca03c945e4037788ade"} Nov 03 20:30:03.621543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:03.621526 4457 scope.go:122] "RemoveContainer" containerID="0ce306148a68de58faad9127327925c6bc1ff93230225ca03c945e4037788ade" Nov 03 20:30:04.638452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.638405 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" event={"ID":"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121","Type":"ContainerStarted","Data":"0164a6705fac619e0742c1b097b71ac71cf973cc6095f9c8b7fe34ab486f0ec3"} Nov 03 20:30:04.641527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.641499 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" event={"ID":"b50409d7-ddee-4335-86eb-34b5b0f87924","Type":"ContainerStarted","Data":"2ed4beb82c345cb8e0e0ac8af185e0b5e459018f410d012ce74058acd2431392"} Nov 03 20:30:04.644102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.644075 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" event={"ID":"ce8d2d9a-946c-4e76-b319-c7d653ac2306","Type":"ContainerStarted","Data":"c34dfa161c1587285bb0676ef4c95e50ca7d56747b57e93b0e6f0f6a800ec909"} Nov 03 20:30:04.646795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.646751 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" event={"ID":"e68e63cf-3ae5-4035-b096-b8d0facc69c0","Type":"ContainerStarted","Data":"33203e00d65946af716012bd7e98daccd4f49f2c91e00cbb40c33b5d94de210e"} Nov 03 20:30:04.649292 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.649270 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" event={"ID":"d168551a-ced0-4313-83f4-6f1d4f3b758d","Type":"ContainerStarted","Data":"0a04b8d44713dc2e7279c052fa5bbcf87f428ff33c4a7d482f4afc29542dcf28"} Nov 03 20:30:04.651152 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.651130 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" event={"ID":"eea2c970-9b94-45c7-b8e7-e1263a11f8f1","Type":"ContainerStarted","Data":"4fae8be0ff2f40fe0e0652e69439f03654d85422af8cd9cc6b59bec39ef39608"} Nov 03 20:30:04.653035 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.653010 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" event={"ID":"6cad965b-7c57-4560-921e-0e5ed244784a","Type":"ContainerStarted","Data":"747750fffc79936404724194aee932223acf43aad091766ccc6d7703f2f740dc"} Nov 03 20:30:04.654894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.654875 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" event={"ID":"c9fce91d-067e-413a-8d5b-85d5b3d04e80","Type":"ContainerStarted","Data":"4162d886b9ffd82c7deb04cc75e03ca54abba2b1b551edbeee07a5e40de0226f"} Nov 03 20:30:04.661497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.661473 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" event={"ID":"f52a0620-190b-418a-afdd-8c8ac053a1d4","Type":"ContainerStarted","Data":"42c908d2f4c25d7365a5d68b18f6adadb2db481502ae8a3cdd32206ac34db98a"} Nov 03 20:30:04.664708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.664657 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" podStartSLOduration=4.664642499 podStartE2EDuration="4.664642499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.660704956 +0000 UTC m=+1983.036301704" watchObservedRunningTime="2026-11-03 20:30:04.664642499 +0000 UTC m=+1983.040239247" Nov 03 20:30:04.664856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.664835 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" event={"ID":"541df3aa-7e72-4bfc-a895-b4c04ea1247e","Type":"ContainerStarted","Data":"35be25c0f1de1246e66c00b711bf165ecf3e891981f8ad16ea7b2c1ddd07fe48"} Nov 03 20:30:04.668528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.668497 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" event={"ID":"ff06a2f7-0660-4ffb-83c8-f362e3e89658","Type":"ContainerStarted","Data":"27bead0ee43aa601d0fe513d06e27eb081c5d80afc0de590a154dc2f2b833e4e"} Nov 03 20:30:04.720563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.720502 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" podStartSLOduration=4.720485329 podStartE2EDuration="4.720485329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.714814101 +0000 UTC m=+1983.090410847" watchObservedRunningTime="2026-11-03 20:30:04.720485329 +0000 UTC m=+1983.096082077" Nov 03 20:30:04.738417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.738356 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" podStartSLOduration=4.738341846 podStartE2EDuration="4.738341846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.733908859 +0000 UTC m=+1983.109505620" watchObservedRunningTime="2026-11-03 20:30:04.738341846 +0000 UTC m=+1983.113938593" Nov 03 20:30:04.770542 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.770492 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" podStartSLOduration=4.770478396 podStartE2EDuration="4.770478396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.754764343 +0000 UTC m=+1983.130361093" watchObservedRunningTime="2026-11-03 20:30:04.770478396 +0000 UTC m=+1983.146075142" Nov 03 20:30:04.783209 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.783152 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" podStartSLOduration=4.783136364 podStartE2EDuration="4.783136364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.778705553 +0000 UTC m=+1983.154302300" watchObservedRunningTime="2026-11-03 20:30:04.783136364 +0000 UTC m=+1983.158733115" Nov 03 20:30:04.814965 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.814910 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" podStartSLOduration=4.814895824 podStartE2EDuration="4.814895824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.807265538 +0000 UTC m=+1983.182862289" watchObservedRunningTime="2026-11-03 20:30:04.814895824 +0000 UTC m=+1983.190492613" Nov 03 20:30:04.831628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.831579 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" podStartSLOduration=4.8315662029999995 podStartE2EDuration="4.831566203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.827120824 +0000 UTC m=+1983.202717572" watchObservedRunningTime="2026-11-03 20:30:04.831566203 +0000 UTC m=+1983.207162951" Nov 03 20:30:04.852566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:04.852514 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" podStartSLOduration=4.852501156 podStartE2EDuration="4.852501156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:04.847514192 +0000 UTC m=+1983.223110936" watchObservedRunningTime="2026-11-03 20:30:04.852501156 +0000 UTC m=+1983.228097947" Nov 03 20:30:05.683466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.683433 4457 generic.go:356] "Generic (PLEG): container finished" podID="b50409d7-ddee-4335-86eb-34b5b0f87924" containerID="2ed4beb82c345cb8e0e0ac8af185e0b5e459018f410d012ce74058acd2431392" exitCode=60 Nov 03 20:30:05.683976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.683546 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" event={"ID":"b50409d7-ddee-4335-86eb-34b5b0f87924","Type":"ContainerDied","Data":"2ed4beb82c345cb8e0e0ac8af185e0b5e459018f410d012ce74058acd2431392"} Nov 03 20:30:05.683976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.683607 4457 scope.go:122] "RemoveContainer" containerID="b2a4aa199dc2567526084c39ffb80ef8a7044d7e832ec1f89ebc451ba54f0277" Nov 03 20:30:05.683976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.683942 4457 scope.go:122] "RemoveContainer" containerID="2ed4beb82c345cb8e0e0ac8af185e0b5e459018f410d012ce74058acd2431392" Nov 03 20:30:05.684546 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:05.684479 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n_odh-ai-gateway-infra(b50409d7-ddee-4335-86eb-34b5b0f87924)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" podUID="b50409d7-ddee-4335-86eb-34b5b0f87924" Nov 03 20:30:05.686086 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.686016 4457 generic.go:356] "Generic (PLEG): container finished" podID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" containerID="c34dfa161c1587285bb0676ef4c95e50ca7d56747b57e93b0e6f0f6a800ec909" exitCode=60 Nov 03 20:30:05.686086 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.686059 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" event={"ID":"ce8d2d9a-946c-4e76-b319-c7d653ac2306","Type":"ContainerDied","Data":"c34dfa161c1587285bb0676ef4c95e50ca7d56747b57e93b0e6f0f6a800ec909"} Nov 03 20:30:05.686525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.686502 4457 scope.go:122] "RemoveContainer" containerID="c34dfa161c1587285bb0676ef4c95e50ca7d56747b57e93b0e6f0f6a800ec909" Nov 03 20:30:05.686980 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:05.686949 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5_odh-ai-gateway-infra(ce8d2d9a-946c-4e76-b319-c7d653ac2306)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" podUID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" Nov 03 20:30:05.688718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.688694 4457 generic.go:356] "Generic (PLEG): container finished" podID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" containerID="33203e00d65946af716012bd7e98daccd4f49f2c91e00cbb40c33b5d94de210e" exitCode=60 Nov 03 20:30:05.688829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.688778 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" event={"ID":"e68e63cf-3ae5-4035-b096-b8d0facc69c0","Type":"ContainerDied","Data":"33203e00d65946af716012bd7e98daccd4f49f2c91e00cbb40c33b5d94de210e"} Nov 03 20:30:05.689138 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.689117 4457 scope.go:122] "RemoveContainer" containerID="33203e00d65946af716012bd7e98daccd4f49f2c91e00cbb40c33b5d94de210e" Nov 03 20:30:05.689576 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:05.689552 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7_odh-ai-gateway-infra(e68e63cf-3ae5-4035-b096-b8d0facc69c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" podUID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" Nov 03 20:30:05.694817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.691990 4457 generic.go:356] "Generic (PLEG): container finished" podID="d168551a-ced0-4313-83f4-6f1d4f3b758d" containerID="0a04b8d44713dc2e7279c052fa5bbcf87f428ff33c4a7d482f4afc29542dcf28" exitCode=60 Nov 03 20:30:05.694817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.692142 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" event={"ID":"d168551a-ced0-4313-83f4-6f1d4f3b758d","Type":"ContainerDied","Data":"0a04b8d44713dc2e7279c052fa5bbcf87f428ff33c4a7d482f4afc29542dcf28"} Nov 03 20:30:05.694817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.692717 4457 scope.go:122] "RemoveContainer" containerID="0a04b8d44713dc2e7279c052fa5bbcf87f428ff33c4a7d482f4afc29542dcf28" Nov 03 20:30:05.694817 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:05.693083 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp_odh-ai-gateway-infra(d168551a-ced0-4313-83f4-6f1d4f3b758d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" podUID="d168551a-ced0-4313-83f4-6f1d4f3b758d" Nov 03 20:30:05.695214 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.695192 4457 generic.go:356] "Generic (PLEG): container finished" podID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" containerID="4fae8be0ff2f40fe0e0652e69439f03654d85422af8cd9cc6b59bec39ef39608" exitCode=60 Nov 03 20:30:05.695292 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.695276 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" event={"ID":"eea2c970-9b94-45c7-b8e7-e1263a11f8f1","Type":"ContainerDied","Data":"4fae8be0ff2f40fe0e0652e69439f03654d85422af8cd9cc6b59bec39ef39608"} Nov 03 20:30:05.695689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.695669 4457 scope.go:122] "RemoveContainer" containerID="4fae8be0ff2f40fe0e0652e69439f03654d85422af8cd9cc6b59bec39ef39608" Nov 03 20:30:05.695979 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:05.695954 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl_odh-ai-gateway-infra(eea2c970-9b94-45c7-b8e7-e1263a11f8f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" podUID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" Nov 03 20:30:05.697679 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.697650 4457 generic.go:356] "Generic (PLEG): container finished" podID="6cad965b-7c57-4560-921e-0e5ed244784a" containerID="747750fffc79936404724194aee932223acf43aad091766ccc6d7703f2f740dc" exitCode=60 Nov 03 20:30:05.697764 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.697679 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" event={"ID":"6cad965b-7c57-4560-921e-0e5ed244784a","Type":"ContainerDied","Data":"747750fffc79936404724194aee932223acf43aad091766ccc6d7703f2f740dc"} Nov 03 20:30:05.698172 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.698155 4457 scope.go:122] "RemoveContainer" containerID="747750fffc79936404724194aee932223acf43aad091766ccc6d7703f2f740dc" Nov 03 20:30:05.698469 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:05.698447 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv_odh-ai-gateway-infra(6cad965b-7c57-4560-921e-0e5ed244784a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" podUID="6cad965b-7c57-4560-921e-0e5ed244784a" Nov 03 20:30:05.700164 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.700142 4457 generic.go:356] "Generic (PLEG): container finished" podID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" containerID="4162d886b9ffd82c7deb04cc75e03ca54abba2b1b551edbeee07a5e40de0226f" exitCode=60 Nov 03 20:30:05.700246 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.700233 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" event={"ID":"c9fce91d-067e-413a-8d5b-85d5b3d04e80","Type":"ContainerDied","Data":"4162d886b9ffd82c7deb04cc75e03ca54abba2b1b551edbeee07a5e40de0226f"} Nov 03 20:30:05.700485 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.700463 4457 scope.go:122] "RemoveContainer" containerID="4162d886b9ffd82c7deb04cc75e03ca54abba2b1b551edbeee07a5e40de0226f" Nov 03 20:30:05.700777 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:05.700755 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm_odh-ai-gateway-infra(c9fce91d-067e-413a-8d5b-85d5b3d04e80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" podUID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" Nov 03 20:30:05.703474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.703338 4457 generic.go:356] "Generic (PLEG): container finished" podID="f52a0620-190b-418a-afdd-8c8ac053a1d4" containerID="42c908d2f4c25d7365a5d68b18f6adadb2db481502ae8a3cdd32206ac34db98a" exitCode=60 Nov 03 20:30:05.703474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.703470 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" event={"ID":"f52a0620-190b-418a-afdd-8c8ac053a1d4","Type":"ContainerDied","Data":"42c908d2f4c25d7365a5d68b18f6adadb2db481502ae8a3cdd32206ac34db98a"} Nov 03 20:30:05.703767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.703750 4457 scope.go:122] "RemoveContainer" containerID="42c908d2f4c25d7365a5d68b18f6adadb2db481502ae8a3cdd32206ac34db98a" Nov 03 20:30:05.703974 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:05.703955 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr_odh-ai-gateway-infra(f52a0620-190b-418a-afdd-8c8ac053a1d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" podUID="f52a0620-190b-418a-afdd-8c8ac053a1d4" Nov 03 20:30:05.705173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.705157 4457 scope.go:122] "RemoveContainer" containerID="0490ff09607bb50a0a95119004a3b468a081bd6f8c82c55b0f5b4a720a0c5408" Nov 03 20:30:05.706487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.706463 4457 generic.go:356] "Generic (PLEG): container finished" podID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" containerID="35be25c0f1de1246e66c00b711bf165ecf3e891981f8ad16ea7b2c1ddd07fe48" exitCode=60 Nov 03 20:30:05.706586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.706545 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" event={"ID":"541df3aa-7e72-4bfc-a895-b4c04ea1247e","Type":"ContainerDied","Data":"35be25c0f1de1246e66c00b711bf165ecf3e891981f8ad16ea7b2c1ddd07fe48"} Nov 03 20:30:05.706745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.706729 4457 scope.go:122] "RemoveContainer" containerID="35be25c0f1de1246e66c00b711bf165ecf3e891981f8ad16ea7b2c1ddd07fe48" Nov 03 20:30:05.707037 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:05.706980 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp_odh-ai-gateway-infra(541df3aa-7e72-4bfc-a895-b4c04ea1247e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" podUID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" Nov 03 20:30:05.708817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.708799 4457 generic.go:356] "Generic (PLEG): container finished" podID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" containerID="27bead0ee43aa601d0fe513d06e27eb081c5d80afc0de590a154dc2f2b833e4e" exitCode=60 Nov 03 20:30:05.708900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.708877 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" event={"ID":"ff06a2f7-0660-4ffb-83c8-f362e3e89658","Type":"ContainerDied","Data":"27bead0ee43aa601d0fe513d06e27eb081c5d80afc0de590a154dc2f2b833e4e"} Nov 03 20:30:05.709115 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.709096 4457 scope.go:122] "RemoveContainer" containerID="27bead0ee43aa601d0fe513d06e27eb081c5d80afc0de590a154dc2f2b833e4e" Nov 03 20:30:05.709422 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:05.709366 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc_odh-ai-gateway-infra(ff06a2f7-0660-4ffb-83c8-f362e3e89658)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" podUID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" Nov 03 20:30:05.710939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.710916 4457 generic.go:356] "Generic (PLEG): container finished" podID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" containerID="0164a6705fac619e0742c1b097b71ac71cf973cc6095f9c8b7fe34ab486f0ec3" exitCode=60 Nov 03 20:30:05.711014 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.710951 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" event={"ID":"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121","Type":"ContainerDied","Data":"0164a6705fac619e0742c1b097b71ac71cf973cc6095f9c8b7fe34ab486f0ec3"} Nov 03 20:30:05.711169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.711155 4457 scope.go:122] "RemoveContainer" containerID="0164a6705fac619e0742c1b097b71ac71cf973cc6095f9c8b7fe34ab486f0ec3" Nov 03 20:30:05.711413 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:05.711380 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k_odh-ai-gateway-infra(c1c5f7ba-2be4-4ca4-9223-7923c9ad9121)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" podUID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" Nov 03 20:30:05.721286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.721266 4457 scope.go:122] "RemoveContainer" containerID="0ce306148a68de58faad9127327925c6bc1ff93230225ca03c945e4037788ade" Nov 03 20:30:05.739000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.738971 4457 scope.go:122] "RemoveContainer" containerID="1169d6f533496a561bbdd7c98a1776a3daebdeeb323ecb4676bd5c010a6d2103" Nov 03 20:30:05.758416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.758377 4457 scope.go:122] "RemoveContainer" containerID="c62893417b2af1014b7ecc51894ce8132ddc7f2a3103d58a321891252ad702f1" Nov 03 20:30:05.797189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.797159 4457 scope.go:122] "RemoveContainer" containerID="7bed30139225ebcadf157f3529bb0e94dc991fbbe7a353e531ccdb87ba02549d" Nov 03 20:30:05.816582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.816553 4457 scope.go:122] "RemoveContainer" containerID="a3469e4dbeb0947f4346edd0efb20510f044a87a1b90170dbb96ee053b7f3760" Nov 03 20:30:05.831614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.831592 4457 scope.go:122] "RemoveContainer" containerID="a23ceccad786e9ba08ebb988bb6902adadb622414b18412ce376f1b651cc86b6" Nov 03 20:30:05.852925 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.851632 4457 scope.go:122] "RemoveContainer" containerID="9e54fa2154a7d83dda87e6192087485dffa0783c735f1d52300971213165e7e3" Nov 03 20:30:05.883276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.883250 4457 scope.go:122] "RemoveContainer" containerID="2d97d019635f9773c674193bc6b908b89477fd6fc13eec0f99869e017a9b21db" Nov 03 20:30:05.901983 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:05.901957 4457 scope.go:122] "RemoveContainer" containerID="430f8b7ebd97d9a8e4056be8e6d7aed19b479c585aebda4ecb01a085dc91fdbe" Nov 03 20:30:06.162642 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:06.162598 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:30:06.723312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:06.723277 4457 scope.go:122] "RemoveContainer" containerID="4fae8be0ff2f40fe0e0652e69439f03654d85422af8cd9cc6b59bec39ef39608" Nov 03 20:30:06.723790 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:06.723707 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl_odh-ai-gateway-infra(eea2c970-9b94-45c7-b8e7-e1263a11f8f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" podUID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" Nov 03 20:30:06.725102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:06.725073 4457 scope.go:122] "RemoveContainer" containerID="747750fffc79936404724194aee932223acf43aad091766ccc6d7703f2f740dc" Nov 03 20:30:06.725447 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:06.725423 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv_odh-ai-gateway-infra(6cad965b-7c57-4560-921e-0e5ed244784a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" podUID="6cad965b-7c57-4560-921e-0e5ed244784a" Nov 03 20:30:06.726680 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:06.726661 4457 scope.go:122] "RemoveContainer" containerID="4162d886b9ffd82c7deb04cc75e03ca54abba2b1b551edbeee07a5e40de0226f" Nov 03 20:30:06.726935 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:06.726915 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm_odh-ai-gateway-infra(c9fce91d-067e-413a-8d5b-85d5b3d04e80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" podUID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" Nov 03 20:30:06.728551 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:06.728531 4457 scope.go:122] "RemoveContainer" containerID="42c908d2f4c25d7365a5d68b18f6adadb2db481502ae8a3cdd32206ac34db98a" Nov 03 20:30:06.728780 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:06.728759 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr_odh-ai-gateway-infra(f52a0620-190b-418a-afdd-8c8ac053a1d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" podUID="f52a0620-190b-418a-afdd-8c8ac053a1d4" Nov 03 20:30:06.730316 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:06.730297 4457 scope.go:122] "RemoveContainer" containerID="35be25c0f1de1246e66c00b711bf165ecf3e891981f8ad16ea7b2c1ddd07fe48" Nov 03 20:30:06.730607 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:06.730583 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp_odh-ai-gateway-infra(541df3aa-7e72-4bfc-a895-b4c04ea1247e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" podUID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" Nov 03 20:30:06.732164 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:06.732146 4457 scope.go:122] "RemoveContainer" containerID="27bead0ee43aa601d0fe513d06e27eb081c5d80afc0de590a154dc2f2b833e4e" Nov 03 20:30:06.732464 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:06.732441 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc_odh-ai-gateway-infra(ff06a2f7-0660-4ffb-83c8-f362e3e89658)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" podUID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" Nov 03 20:30:06.733759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:06.733745 4457 scope.go:122] "RemoveContainer" containerID="0164a6705fac619e0742c1b097b71ac71cf973cc6095f9c8b7fe34ab486f0ec3" Nov 03 20:30:06.734014 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:06.733992 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k_odh-ai-gateway-infra(c1c5f7ba-2be4-4ca4-9223-7923c9ad9121)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" podUID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" Nov 03 20:30:06.735502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:06.735483 4457 scope.go:122] "RemoveContainer" containerID="2ed4beb82c345cb8e0e0ac8af185e0b5e459018f410d012ce74058acd2431392" Nov 03 20:30:06.735714 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:06.735690 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n_odh-ai-gateway-infra(b50409d7-ddee-4335-86eb-34b5b0f87924)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" podUID="b50409d7-ddee-4335-86eb-34b5b0f87924" Nov 03 20:30:06.737129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:06.737111 4457 scope.go:122] "RemoveContainer" containerID="c34dfa161c1587285bb0676ef4c95e50ca7d56747b57e93b0e6f0f6a800ec909" Nov 03 20:30:06.737366 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:06.737344 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5_odh-ai-gateway-infra(ce8d2d9a-946c-4e76-b319-c7d653ac2306)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" podUID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" Nov 03 20:30:06.738887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:06.738871 4457 scope.go:122] "RemoveContainer" containerID="33203e00d65946af716012bd7e98daccd4f49f2c91e00cbb40c33b5d94de210e" Nov 03 20:30:06.739109 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:06.739086 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7_odh-ai-gateway-infra(e68e63cf-3ae5-4035-b096-b8d0facc69c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" podUID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" Nov 03 20:30:06.740655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:06.740638 4457 scope.go:122] "RemoveContainer" containerID="0a04b8d44713dc2e7279c052fa5bbcf87f428ff33c4a7d482f4afc29542dcf28" Nov 03 20:30:06.740879 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:06.740860 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp_odh-ai-gateway-infra(d168551a-ced0-4313-83f4-6f1d4f3b758d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" podUID="d168551a-ced0-4313-83f4-6f1d4f3b758d" Nov 03 20:30:07.278855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:07.278818 4457 scope.go:122] "RemoveContainer" containerID="938c78e5fd248904c6b7d217fee2fa51bb64397dd0ddf1f51bcd37ccd6286871" Nov 03 20:30:13.338290 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:13.338252 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95"] Nov 03 20:30:13.367623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:13.367575 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r"] Nov 03 20:30:15.750239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:15.750203 4457 scope.go:122] "RemoveContainer" containerID="35be25c0f1de1246e66c00b711bf165ecf3e891981f8ad16ea7b2c1ddd07fe48" Nov 03 20:30:16.757311 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:16.756676 4457 scope.go:122] "RemoveContainer" containerID="2ed4beb82c345cb8e0e0ac8af185e0b5e459018f410d012ce74058acd2431392" Nov 03 20:30:16.773126 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:16.773075 4457 scope.go:122] "RemoveContainer" containerID="33203e00d65946af716012bd7e98daccd4f49f2c91e00cbb40c33b5d94de210e" Nov 03 20:30:16.877612 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:16.877582 4457 generic.go:356] "Generic (PLEG): container finished" podID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" containerID="0ef570d24bbe83df7d3c6a78f61f4d3fb5e9b989bc7a30ccfa833a570a226e5c" exitCode=60 Nov 03 20:30:16.877779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:16.877657 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" event={"ID":"541df3aa-7e72-4bfc-a895-b4c04ea1247e","Type":"ContainerDied","Data":"0ef570d24bbe83df7d3c6a78f61f4d3fb5e9b989bc7a30ccfa833a570a226e5c"} Nov 03 20:30:16.877779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:16.877731 4457 scope.go:122] "RemoveContainer" containerID="35be25c0f1de1246e66c00b711bf165ecf3e891981f8ad16ea7b2c1ddd07fe48" Nov 03 20:30:16.878275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:16.878250 4457 scope.go:122] "RemoveContainer" containerID="0ef570d24bbe83df7d3c6a78f61f4d3fb5e9b989bc7a30ccfa833a570a226e5c" Nov 03 20:30:16.878799 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:16.878769 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp_odh-ai-gateway-infra(541df3aa-7e72-4bfc-a895-b4c04ea1247e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" podUID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" Nov 03 20:30:17.750167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:17.750071 4457 scope.go:122] "RemoveContainer" containerID="27bead0ee43aa601d0fe513d06e27eb081c5d80afc0de590a154dc2f2b833e4e" Nov 03 20:30:17.750317 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:17.750258 4457 scope.go:122] "RemoveContainer" containerID="747750fffc79936404724194aee932223acf43aad091766ccc6d7703f2f740dc" Nov 03 20:30:17.750367 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:17.750354 4457 scope.go:122] "RemoveContainer" containerID="0164a6705fac619e0742c1b097b71ac71cf973cc6095f9c8b7fe34ab486f0ec3" Nov 03 20:30:17.894540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:17.894513 4457 generic.go:356] "Generic (PLEG): container finished" podID="b50409d7-ddee-4335-86eb-34b5b0f87924" containerID="026efdbdab32bf96acf71c58ae859fa36455e3a9910990a6c3b36cb39406aaf3" exitCode=60 Nov 03 20:30:17.894940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:17.894586 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" event={"ID":"b50409d7-ddee-4335-86eb-34b5b0f87924","Type":"ContainerDied","Data":"026efdbdab32bf96acf71c58ae859fa36455e3a9910990a6c3b36cb39406aaf3"} Nov 03 20:30:17.894940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:17.894670 4457 scope.go:122] "RemoveContainer" containerID="2ed4beb82c345cb8e0e0ac8af185e0b5e459018f410d012ce74058acd2431392" Nov 03 20:30:17.895147 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:17.895132 4457 scope.go:122] "RemoveContainer" containerID="026efdbdab32bf96acf71c58ae859fa36455e3a9910990a6c3b36cb39406aaf3" Nov 03 20:30:17.895612 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:17.895585 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n_odh-ai-gateway-infra(b50409d7-ddee-4335-86eb-34b5b0f87924)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" podUID="b50409d7-ddee-4335-86eb-34b5b0f87924" Nov 03 20:30:17.897017 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:17.896999 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" event={"ID":"e68e63cf-3ae5-4035-b096-b8d0facc69c0","Type":"ContainerStarted","Data":"46548dca877357667e75c0d28eed2ada0172f8c4e67c1d71f1b6036efe9e47e0"} Nov 03 20:30:17.908853 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:17.908830 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp"] Nov 03 20:30:18.284593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.284562 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" Nov 03 20:30:18.466565 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.466535 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/541df3aa-7e72-4bfc-a895-b4c04ea1247e-kube-api-access-qrx8d\" (UniqueName: \"kubernetes.io/projected/541df3aa-7e72-4bfc-a895-b4c04ea1247e-kube-api-access-qrx8d\") pod \"541df3aa-7e72-4bfc-a895-b4c04ea1247e\" (UID: \"541df3aa-7e72-4bfc-a895-b4c04ea1247e\") " Nov 03 20:30:18.466697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.466677 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/541df3aa-7e72-4bfc-a895-b4c04ea1247e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541df3aa-7e72-4bfc-a895-b4c04ea1247e-maas-api-service-ca\") pod \"541df3aa-7e72-4bfc-a895-b4c04ea1247e\" (UID: \"541df3aa-7e72-4bfc-a895-b4c04ea1247e\") " Nov 03 20:30:18.471946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.470723 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541df3aa-7e72-4bfc-a895-b4c04ea1247e-maas-api-service-ca" pod "541df3aa-7e72-4bfc-a895-b4c04ea1247e" (UID: "541df3aa-7e72-4bfc-a895-b4c04ea1247e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:18.482599 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.482557 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/541df3aa-7e72-4bfc-a895-b4c04ea1247e-kube-api-access-qrx8d" pod "541df3aa-7e72-4bfc-a895-b4c04ea1247e" (UID: "541df3aa-7e72-4bfc-a895-b4c04ea1247e"). InnerVolumeSpecName "kube-api-access-qrx8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:18.571701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.571663 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrx8d\" (UniqueName: \"kubernetes.io/projected/541df3aa-7e72-4bfc-a895-b4c04ea1247e-kube-api-access-qrx8d\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.571701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.571695 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541df3aa-7e72-4bfc-a895-b4c04ea1247e-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:18.750797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.750699 4457 scope.go:122] "RemoveContainer" containerID="42c908d2f4c25d7365a5d68b18f6adadb2db481502ae8a3cdd32206ac34db98a" Nov 03 20:30:18.751328 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.751300 4457 scope.go:122] "RemoveContainer" containerID="4fae8be0ff2f40fe0e0652e69439f03654d85422af8cd9cc6b59bec39ef39608" Nov 03 20:30:18.751525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.751488 4457 scope.go:122] "RemoveContainer" containerID="4162d886b9ffd82c7deb04cc75e03ca54abba2b1b551edbeee07a5e40de0226f" Nov 03 20:30:18.752580 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.751639 4457 scope.go:122] "RemoveContainer" containerID="c34dfa161c1587285bb0676ef4c95e50ca7d56747b57e93b0e6f0f6a800ec909" Nov 03 20:30:18.752580 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.751774 4457 scope.go:122] "RemoveContainer" containerID="0a04b8d44713dc2e7279c052fa5bbcf87f428ff33c4a7d482f4afc29542dcf28" Nov 03 20:30:18.837467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.827521 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg"] Nov 03 20:30:18.943001 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.942045 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" event={"ID":"ff06a2f7-0660-4ffb-83c8-f362e3e89658","Type":"ContainerStarted","Data":"efca346ba643074cabecd4831912118c654e499dd3c01ac8815f997b0929436e"} Nov 03 20:30:18.950050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.949118 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n"] Nov 03 20:30:18.950050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.949181 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" event={"ID":"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121","Type":"ContainerStarted","Data":"649829b918ffa9f52591b3db03cb8329a518de56693f8be534b32f19bb6b27b7"} Nov 03 20:30:18.956451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.956002 4457 generic.go:356] "Generic (PLEG): container finished" podID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" containerID="46548dca877357667e75c0d28eed2ada0172f8c4e67c1d71f1b6036efe9e47e0" exitCode=60 Nov 03 20:30:18.956451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.956300 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" event={"ID":"e68e63cf-3ae5-4035-b096-b8d0facc69c0","Type":"ContainerDied","Data":"46548dca877357667e75c0d28eed2ada0172f8c4e67c1d71f1b6036efe9e47e0"} Nov 03 20:30:18.956451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.956352 4457 scope.go:122] "RemoveContainer" containerID="33203e00d65946af716012bd7e98daccd4f49f2c91e00cbb40c33b5d94de210e" Nov 03 20:30:18.958033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.957208 4457 scope.go:122] "RemoveContainer" containerID="46548dca877357667e75c0d28eed2ada0172f8c4e67c1d71f1b6036efe9e47e0" Nov 03 20:30:18.958033 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:18.957684 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7_odh-ai-gateway-infra(e68e63cf-3ae5-4035-b096-b8d0facc69c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" podUID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" Nov 03 20:30:18.970540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.967938 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" event={"ID":"6cad965b-7c57-4560-921e-0e5ed244784a","Type":"ContainerStarted","Data":"977a90dbe5af805764a35ef94986d7bd5bda8ea3a96fb8c2d3e393a4e9260714"} Nov 03 20:30:18.976460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.972517 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" event={"ID":"541df3aa-7e72-4bfc-a895-b4c04ea1247e","Type":"ContainerDied","Data":"9b36393683d2df7b73022320307271c5578f583207c3b0f8259b96fe75ff7862"} Nov 03 20:30:18.976460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.972606 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp" Nov 03 20:30:18.983789 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:18.982097 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7"] Nov 03 20:30:19.007684 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:19.007569 4457 scope.go:122] "RemoveContainer" containerID="0ef570d24bbe83df7d3c6a78f61f4d3fb5e9b989bc7a30ccfa833a570a226e5c" Nov 03 20:30:19.248475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:19.239966 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp"] Nov 03 20:30:19.258795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:19.258713 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3e9176-29895630-4s7fp"] Nov 03 20:30:19.490720 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:19.489555 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" Nov 03 20:30:19.634056 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:19.633987 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50409d7-ddee-4335-86eb-34b5b0f87924-kube-api-access-clfds\" (UniqueName: \"kubernetes.io/projected/b50409d7-ddee-4335-86eb-34b5b0f87924-kube-api-access-clfds\") pod \"b50409d7-ddee-4335-86eb-34b5b0f87924\" (UID: \"b50409d7-ddee-4335-86eb-34b5b0f87924\") " Nov 03 20:30:19.634693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:19.634666 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b50409d7-ddee-4335-86eb-34b5b0f87924-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50409d7-ddee-4335-86eb-34b5b0f87924-maas-api-service-ca\") pod \"b50409d7-ddee-4335-86eb-34b5b0f87924\" (UID: \"b50409d7-ddee-4335-86eb-34b5b0f87924\") " Nov 03 20:30:19.635803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:19.635775 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50409d7-ddee-4335-86eb-34b5b0f87924-maas-api-service-ca" pod "b50409d7-ddee-4335-86eb-34b5b0f87924" (UID: "b50409d7-ddee-4335-86eb-34b5b0f87924"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:19.667659 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:19.667619 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50409d7-ddee-4335-86eb-34b5b0f87924-kube-api-access-clfds" pod "b50409d7-ddee-4335-86eb-34b5b0f87924" (UID: "b50409d7-ddee-4335-86eb-34b5b0f87924"). InnerVolumeSpecName "kube-api-access-clfds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:19.737338 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:19.737302 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clfds\" (UniqueName: \"kubernetes.io/projected/b50409d7-ddee-4335-86eb-34b5b0f87924-kube-api-access-clfds\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:19.737338 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:19.737336 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b50409d7-ddee-4335-86eb-34b5b0f87924-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:19.767098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:19.767072 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" path="/var/lib/kubelet/pods/541df3aa-7e72-4bfc-a895-b4c04ea1247e/volumes" Nov 03 20:30:19.975456 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:19.975361 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc"] Nov 03 20:30:19.987252 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:19.987207 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" event={"ID":"d168551a-ced0-4313-83f4-6f1d4f3b758d","Type":"ContainerStarted","Data":"b28e3fa437e4d70da2d9dd8f9d38ed953cc1e8cdf92017e9f7311314072dc5a1"} Nov 03 20:30:20.000561 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.000514 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" event={"ID":"eea2c970-9b94-45c7-b8e7-e1263a11f8f1","Type":"ContainerStarted","Data":"e868d5de20cfe279fb210f9dfff00232f362181cfeae3e4585594da54aab9189"} Nov 03 20:30:20.003396 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.003350 4457 generic.go:356] "Generic (PLEG): container finished" podID="6cad965b-7c57-4560-921e-0e5ed244784a" containerID="977a90dbe5af805764a35ef94986d7bd5bda8ea3a96fb8c2d3e393a4e9260714" exitCode=60 Nov 03 20:30:20.003617 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.003593 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" event={"ID":"6cad965b-7c57-4560-921e-0e5ed244784a","Type":"ContainerDied","Data":"977a90dbe5af805764a35ef94986d7bd5bda8ea3a96fb8c2d3e393a4e9260714"} Nov 03 20:30:20.003707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.003692 4457 scope.go:122] "RemoveContainer" containerID="747750fffc79936404724194aee932223acf43aad091766ccc6d7703f2f740dc" Nov 03 20:30:20.004183 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.004162 4457 scope.go:122] "RemoveContainer" containerID="977a90dbe5af805764a35ef94986d7bd5bda8ea3a96fb8c2d3e393a4e9260714" Nov 03 20:30:20.004697 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:20.004669 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv_odh-ai-gateway-infra(6cad965b-7c57-4560-921e-0e5ed244784a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" podUID="6cad965b-7c57-4560-921e-0e5ed244784a" Nov 03 20:30:20.009605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.009576 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" event={"ID":"c9fce91d-067e-413a-8d5b-85d5b3d04e80","Type":"ContainerStarted","Data":"f2bfa1fb731f345cd8b4c9e1b31bf8eef11d83798e75ee5a2518181dd11cd2be"} Nov 03 20:30:20.018883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.018851 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" event={"ID":"f52a0620-190b-418a-afdd-8c8ac053a1d4","Type":"ContainerStarted","Data":"b9ac36d9b5acca790b02cd5fcb286bbe3b28d172f4ed958685c0cc49ef902886"} Nov 03 20:30:20.028145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.028116 4457 generic.go:356] "Generic (PLEG): container finished" podID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" containerID="efca346ba643074cabecd4831912118c654e499dd3c01ac8815f997b0929436e" exitCode=60 Nov 03 20:30:20.028308 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.028207 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" event={"ID":"ff06a2f7-0660-4ffb-83c8-f362e3e89658","Type":"ContainerDied","Data":"efca346ba643074cabecd4831912118c654e499dd3c01ac8815f997b0929436e"} Nov 03 20:30:20.031304 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.031277 4457 scope.go:122] "RemoveContainer" containerID="27bead0ee43aa601d0fe513d06e27eb081c5d80afc0de590a154dc2f2b833e4e" Nov 03 20:30:20.031579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.031551 4457 generic.go:356] "Generic (PLEG): container finished" podID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" containerID="649829b918ffa9f52591b3db03cb8329a518de56693f8be534b32f19bb6b27b7" exitCode=60 Nov 03 20:30:20.031665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.031632 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" event={"ID":"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121","Type":"ContainerDied","Data":"649829b918ffa9f52591b3db03cb8329a518de56693f8be534b32f19bb6b27b7"} Nov 03 20:30:20.032064 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.032047 4457 scope.go:122] "RemoveContainer" containerID="649829b918ffa9f52591b3db03cb8329a518de56693f8be534b32f19bb6b27b7" Nov 03 20:30:20.032501 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:20.032475 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k_odh-ai-gateway-infra(c1c5f7ba-2be4-4ca4-9223-7923c9ad9121)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" podUID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" Nov 03 20:30:20.035119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.034734 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" Nov 03 20:30:20.036020 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.034735 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n" event={"ID":"b50409d7-ddee-4335-86eb-34b5b0f87924","Type":"ContainerDied","Data":"b19f8bca27785626dbd4c90e9af761e0767281863902887147553766a747fa3b"} Nov 03 20:30:20.039216 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.039189 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" event={"ID":"ce8d2d9a-946c-4e76-b319-c7d653ac2306","Type":"ContainerStarted","Data":"b376975b35a90c605c1420cdee8d4a373bf1c227c3e3fe03fb0cdf228206a8f2"} Nov 03 20:30:20.060715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.056647 4457 scope.go:122] "RemoveContainer" containerID="0164a6705fac619e0742c1b097b71ac71cf973cc6095f9c8b7fe34ab486f0ec3" Nov 03 20:30:20.072665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.072633 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k"] Nov 03 20:30:20.113322 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.113286 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv"] Nov 03 20:30:20.134458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.134433 4457 scope.go:122] "RemoveContainer" containerID="026efdbdab32bf96acf71c58ae859fa36455e3a9910990a6c3b36cb39406aaf3" Nov 03 20:30:20.136728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.135829 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n"] Nov 03 20:30:20.152634 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.151255 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-d412cf-29895630-qwd5n"] Nov 03 20:30:20.314545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.314518 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" Nov 03 20:30:20.460174 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.460102 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff06a2f7-0660-4ffb-83c8-f362e3e89658-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff06a2f7-0660-4ffb-83c8-f362e3e89658-maas-api-service-ca\") pod \"ff06a2f7-0660-4ffb-83c8-f362e3e89658\" (UID: \"ff06a2f7-0660-4ffb-83c8-f362e3e89658\") " Nov 03 20:30:20.460365 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.460339 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff06a2f7-0660-4ffb-83c8-f362e3e89658-kube-api-access-2rcjn\" (UniqueName: \"kubernetes.io/projected/ff06a2f7-0660-4ffb-83c8-f362e3e89658-kube-api-access-2rcjn\") pod \"ff06a2f7-0660-4ffb-83c8-f362e3e89658\" (UID: \"ff06a2f7-0660-4ffb-83c8-f362e3e89658\") " Nov 03 20:30:20.461724 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.461247 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff06a2f7-0660-4ffb-83c8-f362e3e89658-maas-api-service-ca" pod "ff06a2f7-0660-4ffb-83c8-f362e3e89658" (UID: "ff06a2f7-0660-4ffb-83c8-f362e3e89658"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.462534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.462513 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff06a2f7-0660-4ffb-83c8-f362e3e89658-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.463728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.463699 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff06a2f7-0660-4ffb-83c8-f362e3e89658-kube-api-access-2rcjn" pod "ff06a2f7-0660-4ffb-83c8-f362e3e89658" (UID: "ff06a2f7-0660-4ffb-83c8-f362e3e89658"). InnerVolumeSpecName "kube-api-access-2rcjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.564362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.564328 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rcjn\" (UniqueName: \"kubernetes.io/projected/ff06a2f7-0660-4ffb-83c8-f362e3e89658-kube-api-access-2rcjn\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.573167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.573141 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" Nov 03 20:30:20.690984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.690920 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e68e63cf-3ae5-4035-b096-b8d0facc69c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68e63cf-3ae5-4035-b096-b8d0facc69c0-maas-api-service-ca\") pod \"e68e63cf-3ae5-4035-b096-b8d0facc69c0\" (UID: \"e68e63cf-3ae5-4035-b096-b8d0facc69c0\") " Nov 03 20:30:20.691285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.691254 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68e63cf-3ae5-4035-b096-b8d0facc69c0-kube-api-access-c9wms\" (UniqueName: \"kubernetes.io/projected/e68e63cf-3ae5-4035-b096-b8d0facc69c0-kube-api-access-c9wms\") pod \"e68e63cf-3ae5-4035-b096-b8d0facc69c0\" (UID: \"e68e63cf-3ae5-4035-b096-b8d0facc69c0\") " Nov 03 20:30:20.693793 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.693763 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68e63cf-3ae5-4035-b096-b8d0facc69c0-maas-api-service-ca" pod "e68e63cf-3ae5-4035-b096-b8d0facc69c0" (UID: "e68e63cf-3ae5-4035-b096-b8d0facc69c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:20.695591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.695564 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68e63cf-3ae5-4035-b096-b8d0facc69c0-kube-api-access-c9wms" pod "e68e63cf-3ae5-4035-b096-b8d0facc69c0" (UID: "e68e63cf-3ae5-4035-b096-b8d0facc69c0"). InnerVolumeSpecName "kube-api-access-c9wms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:20.794614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.794532 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68e63cf-3ae5-4035-b096-b8d0facc69c0-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:20.794614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:20.794561 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9wms\" (UniqueName: \"kubernetes.io/projected/e68e63cf-3ae5-4035-b096-b8d0facc69c0-kube-api-access-c9wms\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.025974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.025941 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp"] Nov 03 20:30:21.047680 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.047647 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm"] Nov 03 20:30:21.065766 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.065174 4457 generic.go:356] "Generic (PLEG): container finished" podID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" containerID="e868d5de20cfe279fb210f9dfff00232f362181cfeae3e4585594da54aab9189" exitCode=60 Nov 03 20:30:21.065766 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.065363 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" event={"ID":"eea2c970-9b94-45c7-b8e7-e1263a11f8f1","Type":"ContainerDied","Data":"e868d5de20cfe279fb210f9dfff00232f362181cfeae3e4585594da54aab9189"} Nov 03 20:30:21.065766 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.065510 4457 scope.go:122] "RemoveContainer" containerID="4fae8be0ff2f40fe0e0652e69439f03654d85422af8cd9cc6b59bec39ef39608" Nov 03 20:30:21.066439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.066419 4457 scope.go:122] "RemoveContainer" containerID="e868d5de20cfe279fb210f9dfff00232f362181cfeae3e4585594da54aab9189" Nov 03 20:30:21.066935 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:21.066910 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl_odh-ai-gateway-infra(eea2c970-9b94-45c7-b8e7-e1263a11f8f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" podUID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" Nov 03 20:30:21.073055 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.073034 4457 generic.go:356] "Generic (PLEG): container finished" podID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" containerID="f2bfa1fb731f345cd8b4c9e1b31bf8eef11d83798e75ee5a2518181dd11cd2be" exitCode=60 Nov 03 20:30:21.073294 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.073276 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" event={"ID":"c9fce91d-067e-413a-8d5b-85d5b3d04e80","Type":"ContainerDied","Data":"f2bfa1fb731f345cd8b4c9e1b31bf8eef11d83798e75ee5a2518181dd11cd2be"} Nov 03 20:30:21.076608 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.076584 4457 generic.go:356] "Generic (PLEG): container finished" podID="f52a0620-190b-418a-afdd-8c8ac053a1d4" containerID="b9ac36d9b5acca790b02cd5fcb286bbe3b28d172f4ed958685c0cc49ef902886" exitCode=60 Nov 03 20:30:21.076709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.076656 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" event={"ID":"f52a0620-190b-418a-afdd-8c8ac053a1d4","Type":"ContainerDied","Data":"b9ac36d9b5acca790b02cd5fcb286bbe3b28d172f4ed958685c0cc49ef902886"} Nov 03 20:30:21.076771 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.076736 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl"] Nov 03 20:30:21.077194 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.077157 4457 scope.go:122] "RemoveContainer" containerID="b9ac36d9b5acca790b02cd5fcb286bbe3b28d172f4ed958685c0cc49ef902886" Nov 03 20:30:21.077767 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:21.077634 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr_odh-ai-gateway-infra(f52a0620-190b-418a-afdd-8c8ac053a1d4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" podUID="f52a0620-190b-418a-afdd-8c8ac053a1d4" Nov 03 20:30:21.079688 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.079592 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" event={"ID":"ff06a2f7-0660-4ffb-83c8-f362e3e89658","Type":"ContainerDied","Data":"1877d3ff6e2b19c2b7066297d650bbaad270b9e6a1be7b13ce00a9f21a9bad38"} Nov 03 20:30:21.079776 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.079688 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc" Nov 03 20:30:21.089121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.089096 4457 scope.go:122] "RemoveContainer" containerID="4162d886b9ffd82c7deb04cc75e03ca54abba2b1b551edbeee07a5e40de0226f" Nov 03 20:30:21.094194 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.093051 4457 generic.go:356] "Generic (PLEG): container finished" podID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" containerID="b376975b35a90c605c1420cdee8d4a373bf1c227c3e3fe03fb0cdf228206a8f2" exitCode=60 Nov 03 20:30:21.094194 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.093149 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" event={"ID":"ce8d2d9a-946c-4e76-b319-c7d653ac2306","Type":"ContainerDied","Data":"b376975b35a90c605c1420cdee8d4a373bf1c227c3e3fe03fb0cdf228206a8f2"} Nov 03 20:30:21.094194 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.093649 4457 scope.go:122] "RemoveContainer" containerID="b376975b35a90c605c1420cdee8d4a373bf1c227c3e3fe03fb0cdf228206a8f2" Nov 03 20:30:21.094194 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:21.094108 4457 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5_odh-ai-gateway-infra(ce8d2d9a-946c-4e76-b319-c7d653ac2306)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" podUID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" Nov 03 20:30:21.098694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.096303 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" event={"ID":"e68e63cf-3ae5-4035-b096-b8d0facc69c0","Type":"ContainerDied","Data":"a15b999dfe926701399dc4bf5ddc5abf548d2cd7a3a2963ce1f80491c6b92646"} Nov 03 20:30:21.098694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.096442 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7" Nov 03 20:30:21.103763 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.103736 4457 generic.go:356] "Generic (PLEG): container finished" podID="d168551a-ced0-4313-83f4-6f1d4f3b758d" containerID="b28e3fa437e4d70da2d9dd8f9d38ed953cc1e8cdf92017e9f7311314072dc5a1" exitCode=60 Nov 03 20:30:21.103856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.103844 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" event={"ID":"d168551a-ced0-4313-83f4-6f1d4f3b758d","Type":"ContainerDied","Data":"b28e3fa437e4d70da2d9dd8f9d38ed953cc1e8cdf92017e9f7311314072dc5a1"} Nov 03 20:30:21.125780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.125661 4457 scope.go:122] "RemoveContainer" containerID="42c908d2f4c25d7365a5d68b18f6adadb2db481502ae8a3cdd32206ac34db98a" Nov 03 20:30:21.160791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.160767 4457 scope.go:122] "RemoveContainer" containerID="efca346ba643074cabecd4831912118c654e499dd3c01ac8815f997b0929436e" Nov 03 20:30:21.194135 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.193051 4457 scope.go:122] "RemoveContainer" containerID="c34dfa161c1587285bb0676ef4c95e50ca7d56747b57e93b0e6f0f6a800ec909" Nov 03 20:30:21.224844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.224525 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr"] Nov 03 20:30:21.241666 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.241636 4457 scope.go:122] "RemoveContainer" containerID="46548dca877357667e75c0d28eed2ada0172f8c4e67c1d71f1b6036efe9e47e0" Nov 03 20:30:21.262516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.262479 4457 scope.go:122] "RemoveContainer" containerID="0a04b8d44713dc2e7279c052fa5bbcf87f428ff33c4a7d482f4afc29542dcf28" Nov 03 20:30:21.329117 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.329067 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5"] Nov 03 20:30:21.361943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.361877 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" Nov 03 20:30:21.379751 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.379716 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7"] Nov 03 20:30:21.390597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.390533 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-34e844-29895630-nm9d7"] Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.402429 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j"] Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.403862 4457 cpu_manager.go:414] "containerMap: removing container" podUID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.403885 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.403902 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.403910 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.403926 4457 cpu_manager.go:414] "containerMap: removing container" podUID="b50409d7-ddee-4335-86eb-34b5b0f87924" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.403934 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50409d7-ddee-4335-86eb-34b5b0f87924" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.403946 4457 cpu_manager.go:414] "containerMap: removing container" podUID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.403953 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.403974 4457 cpu_manager.go:414] "containerMap: removing container" podUID="6cad965b-7c57-4560-921e-0e5ed244784a" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.403981 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cad965b-7c57-4560-921e-0e5ed244784a" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404003 4457 cpu_manager.go:414] "containerMap: removing container" podUID="6cad965b-7c57-4560-921e-0e5ed244784a" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404010 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cad965b-7c57-4560-921e-0e5ed244784a" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404032 4457 cpu_manager.go:414] "containerMap: removing container" podUID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404040 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404057 4457 cpu_manager.go:414] "containerMap: removing container" podUID="6cad965b-7c57-4560-921e-0e5ed244784a" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404067 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cad965b-7c57-4560-921e-0e5ed244784a" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404099 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404107 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404144 4457 cpu_manager.go:414] "containerMap: removing container" podUID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404151 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404164 4457 cpu_manager.go:414] "containerMap: removing container" podUID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404171 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404185 4457 cpu_manager.go:414] "containerMap: removing container" podUID="b50409d7-ddee-4335-86eb-34b5b0f87924" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404191 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50409d7-ddee-4335-86eb-34b5b0f87924" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404203 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" containerName="cleanup" Nov 03 20:30:21.404411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.404210 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" containerName="cleanup" Nov 03 20:30:21.409880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.407180 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" containerName="cleanup" Nov 03 20:30:21.409880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.407204 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" containerName="cleanup" Nov 03 20:30:21.409880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.407215 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50409d7-ddee-4335-86eb-34b5b0f87924" containerName="cleanup" Nov 03 20:30:21.409880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.407228 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50409d7-ddee-4335-86eb-34b5b0f87924" containerName="cleanup" Nov 03 20:30:21.409880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.407238 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" containerName="cleanup" Nov 03 20:30:21.409880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.407256 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" containerName="cleanup" Nov 03 20:30:21.409880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.407266 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50409d7-ddee-4335-86eb-34b5b0f87924" containerName="cleanup" Nov 03 20:30:21.409880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.407278 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cad965b-7c57-4560-921e-0e5ed244784a" containerName="cleanup" Nov 03 20:30:21.409880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.407289 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cad965b-7c57-4560-921e-0e5ed244784a" containerName="cleanup" Nov 03 20:30:21.409880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.407296 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" containerName="cleanup" Nov 03 20:30:21.409880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.407307 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" containerName="cleanup" Nov 03 20:30:21.409880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.407321 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" containerName="cleanup" Nov 03 20:30:21.431676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.431649 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz"] Nov 03 20:30:21.431756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.431698 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" Nov 03 20:30:21.433896 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.433870 4457 cpu_manager.go:414] "containerMap: removing container" podUID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" containerName="cleanup" Nov 03 20:30:21.433896 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.433896 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" containerName="cleanup" Nov 03 20:30:21.434066 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.433922 4457 cpu_manager.go:414] "containerMap: removing container" podUID="b50409d7-ddee-4335-86eb-34b5b0f87924" containerName="cleanup" Nov 03 20:30:21.434066 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.433929 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50409d7-ddee-4335-86eb-34b5b0f87924" containerName="cleanup" Nov 03 20:30:21.434243 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.434226 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cad965b-7c57-4560-921e-0e5ed244784a" containerName="cleanup" Nov 03 20:30:21.434299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.434250 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="541df3aa-7e72-4bfc-a895-b4c04ea1247e" containerName="cleanup" Nov 03 20:30:21.434299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.434268 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" containerName="cleanup" Nov 03 20:30:21.435450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.435419 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:30:21.440796 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.440774 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j"] Nov 03 20:30:21.440896 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.440808 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc"] Nov 03 20:30:21.440896 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.440821 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz"] Nov 03 20:30:21.440896 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.440883 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" Nov 03 20:30:21.448564 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.448482 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-77e83b-29895630-22qhc"] Nov 03 20:30:21.493069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.492724 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" Nov 03 20:30:21.524453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.524415 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cad965b-7c57-4560-921e-0e5ed244784a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cad965b-7c57-4560-921e-0e5ed244784a-maas-api-service-ca\") pod \"6cad965b-7c57-4560-921e-0e5ed244784a\" (UID: \"6cad965b-7c57-4560-921e-0e5ed244784a\") " Nov 03 20:30:21.524762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.524742 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cad965b-7c57-4560-921e-0e5ed244784a-kube-api-access-92n2d\" (UniqueName: \"kubernetes.io/projected/6cad965b-7c57-4560-921e-0e5ed244784a-kube-api-access-92n2d\") pod \"6cad965b-7c57-4560-921e-0e5ed244784a\" (UID: \"6cad965b-7c57-4560-921e-0e5ed244784a\") " Nov 03 20:30:21.525188 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.525165 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tf2t\" (UniqueName: \"kubernetes.io/projected/b1023951-0cef-4afe-b7d8-76d8a9775abb-kube-api-access-2tf2t\") pod \"maas-api-revoke-sub-5ac980d0-x5t5j\" (UID: \"b1023951-0cef-4afe-b7d8-76d8a9775abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" Nov 03 20:30:21.525375 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.525361 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ac980d0-x5t5j\" (UID: \"b1023951-0cef-4afe-b7d8-76d8a9775abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" Nov 03 20:30:21.525504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.525489 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ac980d0-x5t5j\" (UID: \"b1023951-0cef-4afe-b7d8-76d8a9775abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" Nov 03 20:30:21.526212 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.526194 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cad965b-7c57-4560-921e-0e5ed244784a-maas-api-service-ca" pod "6cad965b-7c57-4560-921e-0e5ed244784a" (UID: "6cad965b-7c57-4560-921e-0e5ed244784a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:21.546684 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.540569 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cad965b-7c57-4560-921e-0e5ed244784a-kube-api-access-92n2d" pod "6cad965b-7c57-4560-921e-0e5ed244784a" (UID: "6cad965b-7c57-4560-921e-0e5ed244784a"). InnerVolumeSpecName "kube-api-access-92n2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:21.578599 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.578567 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" Nov 03 20:30:21.583431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.583407 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" Nov 03 20:30:21.627244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.627192 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9fce91d-067e-413a-8d5b-85d5b3d04e80-kube-api-access-sxc8n\" (UniqueName: \"kubernetes.io/projected/c9fce91d-067e-413a-8d5b-85d5b3d04e80-kube-api-access-sxc8n\") pod \"c9fce91d-067e-413a-8d5b-85d5b3d04e80\" (UID: \"c9fce91d-067e-413a-8d5b-85d5b3d04e80\") " Nov 03 20:30:21.627723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.627694 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9fce91d-067e-413a-8d5b-85d5b3d04e80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9fce91d-067e-413a-8d5b-85d5b3d04e80-maas-api-service-ca\") pod \"c9fce91d-067e-413a-8d5b-85d5b3d04e80\" (UID: \"c9fce91d-067e-413a-8d5b-85d5b3d04e80\") " Nov 03 20:30:21.628355 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.628330 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeba7587-8vhhz\" (UID: \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" Nov 03 20:30:21.628520 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.628495 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9fce91d-067e-413a-8d5b-85d5b3d04e80-maas-api-service-ca" pod "c9fce91d-067e-413a-8d5b-85d5b3d04e80" (UID: "c9fce91d-067e-413a-8d5b-85d5b3d04e80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:21.628591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.628568 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzdrd\" (UniqueName: \"kubernetes.io/projected/7f4de79d-18ac-4f57-803e-fb09ddcc8134-kube-api-access-jzdrd\") pod \"maas-api-revoke-sub-aeba7587-8vhhz\" (UID: \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" Nov 03 20:30:21.628804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.628785 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tf2t\" (UniqueName: \"kubernetes.io/projected/b1023951-0cef-4afe-b7d8-76d8a9775abb-kube-api-access-2tf2t\") pod \"maas-api-revoke-sub-5ac980d0-x5t5j\" (UID: \"b1023951-0cef-4afe-b7d8-76d8a9775abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" Nov 03 20:30:21.628871 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.628822 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeba7587-8vhhz\" (UID: \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" Nov 03 20:30:21.628943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.628900 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ac980d0-x5t5j\" (UID: \"b1023951-0cef-4afe-b7d8-76d8a9775abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" Nov 03 20:30:21.628943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.628927 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ac980d0-x5t5j\" (UID: \"b1023951-0cef-4afe-b7d8-76d8a9775abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" Nov 03 20:30:21.629497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.629185 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cad965b-7c57-4560-921e-0e5ed244784a-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.629497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.629207 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92n2d\" (UniqueName: \"kubernetes.io/projected/6cad965b-7c57-4560-921e-0e5ed244784a-kube-api-access-92n2d\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.629497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.629222 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9fce91d-067e-413a-8d5b-85d5b3d04e80-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.629958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.629934 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ac980d0-x5t5j\" (UID: \"b1023951-0cef-4afe-b7d8-76d8a9775abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" Nov 03 20:30:21.630891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.630869 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9fce91d-067e-413a-8d5b-85d5b3d04e80-kube-api-access-sxc8n" pod "c9fce91d-067e-413a-8d5b-85d5b3d04e80" (UID: "c9fce91d-067e-413a-8d5b-85d5b3d04e80"). InnerVolumeSpecName "kube-api-access-sxc8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:21.631803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.631726 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ac980d0-x5t5j\" (UID: \"b1023951-0cef-4afe-b7d8-76d8a9775abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" Nov 03 20:30:21.647140 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.647107 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tf2t\" (UniqueName: \"kubernetes.io/projected/b1023951-0cef-4afe-b7d8-76d8a9775abb-kube-api-access-2tf2t\") pod \"maas-api-revoke-sub-5ac980d0-x5t5j\" (UID: \"b1023951-0cef-4afe-b7d8-76d8a9775abb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" Nov 03 20:30:21.732640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.732497 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-kube-api-access-57hcr\" (UniqueName: \"kubernetes.io/projected/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-kube-api-access-57hcr\") pod \"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121\" (UID: \"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121\") " Nov 03 20:30:21.734815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.732694 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d168551a-ced0-4313-83f4-6f1d4f3b758d-kube-api-access-7zbpg\" (UniqueName: \"kubernetes.io/projected/d168551a-ced0-4313-83f4-6f1d4f3b758d-kube-api-access-7zbpg\") pod \"d168551a-ced0-4313-83f4-6f1d4f3b758d\" (UID: \"d168551a-ced0-4313-83f4-6f1d4f3b758d\") " Nov 03 20:30:21.734815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.732902 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-maas-api-service-ca\") pod \"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121\" (UID: \"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121\") " Nov 03 20:30:21.734815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.733104 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d168551a-ced0-4313-83f4-6f1d4f3b758d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d168551a-ced0-4313-83f4-6f1d4f3b758d-maas-api-service-ca\") pod \"d168551a-ced0-4313-83f4-6f1d4f3b758d\" (UID: \"d168551a-ced0-4313-83f4-6f1d4f3b758d\") " Nov 03 20:30:21.734815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.733608 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeba7587-8vhhz\" (UID: \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" Nov 03 20:30:21.734815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.733975 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeba7587-8vhhz\" (UID: \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" Nov 03 20:30:21.734815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.734051 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzdrd\" (UniqueName: \"kubernetes.io/projected/7f4de79d-18ac-4f57-803e-fb09ddcc8134-kube-api-access-jzdrd\") pod \"maas-api-revoke-sub-aeba7587-8vhhz\" (UID: \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" Nov 03 20:30:21.734815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.734279 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxc8n\" (UniqueName: \"kubernetes.io/projected/c9fce91d-067e-413a-8d5b-85d5b3d04e80-kube-api-access-sxc8n\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.774457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.753653 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d168551a-ced0-4313-83f4-6f1d4f3b758d-maas-api-service-ca" pod "d168551a-ced0-4313-83f4-6f1d4f3b758d" (UID: "d168551a-ced0-4313-83f4-6f1d4f3b758d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:21.774457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.761939 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzdrd\" (UniqueName: \"kubernetes.io/projected/7f4de79d-18ac-4f57-803e-fb09ddcc8134-kube-api-access-jzdrd\") pod \"maas-api-revoke-sub-aeba7587-8vhhz\" (UID: \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" Nov 03 20:30:21.774457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.762237 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-maas-api-service-ca" pod "c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" (UID: "c1c5f7ba-2be4-4ca4-9223-7923c9ad9121"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:21.774457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.764656 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d168551a-ced0-4313-83f4-6f1d4f3b758d-kube-api-access-7zbpg" pod "d168551a-ced0-4313-83f4-6f1d4f3b758d" (UID: "d168551a-ced0-4313-83f4-6f1d4f3b758d"). InnerVolumeSpecName "kube-api-access-7zbpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:21.774457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.765228 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aeba7587-8vhhz\" (UID: \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" Nov 03 20:30:21.774457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.768363 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aeba7587-8vhhz\" (UID: \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" Nov 03 20:30:21.776573 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.776544 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-kube-api-access-57hcr" pod "c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" (UID: "c1c5f7ba-2be4-4ca4-9223-7923c9ad9121"). InnerVolumeSpecName "kube-api-access-57hcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:21.807201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.807161 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b50409d7-ddee-4335-86eb-34b5b0f87924" path="/var/lib/kubelet/pods/b50409d7-ddee-4335-86eb-34b5b0f87924/volumes" Nov 03 20:30:21.808814 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.808783 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68e63cf-3ae5-4035-b096-b8d0facc69c0" path="/var/lib/kubelet/pods/e68e63cf-3ae5-4035-b096-b8d0facc69c0/volumes" Nov 03 20:30:21.809903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.809487 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff06a2f7-0660-4ffb-83c8-f362e3e89658" path="/var/lib/kubelet/pods/ff06a2f7-0660-4ffb-83c8-f362e3e89658/volumes" Nov 03 20:30:21.822160 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.821972 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" Nov 03 20:30:21.836012 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.835977 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" Nov 03 20:30:21.836310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.836078 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.836310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.836308 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d168551a-ced0-4313-83f4-6f1d4f3b758d-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.836482 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.836322 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57hcr\" (UniqueName: \"kubernetes.io/projected/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121-kube-api-access-57hcr\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:21.836482 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:21.836335 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zbpg\" (UniqueName: \"kubernetes.io/projected/d168551a-ced0-4313-83f4-6f1d4f3b758d-kube-api-access-7zbpg\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.118239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.118151 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" Nov 03 20:30:22.118239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.118185 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp" event={"ID":"d168551a-ced0-4313-83f4-6f1d4f3b758d","Type":"ContainerDied","Data":"0ed53dc20b47cd0e586916a61857a903f4b75402647c76f1500f5436ebc142df"} Nov 03 20:30:22.118239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.118239 4457 scope.go:122] "RemoveContainer" containerID="b28e3fa437e4d70da2d9dd8f9d38ed953cc1e8cdf92017e9f7311314072dc5a1" Nov 03 20:30:22.125702 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.125678 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" Nov 03 20:30:22.127252 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.126074 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv" event={"ID":"6cad965b-7c57-4560-921e-0e5ed244784a","Type":"ContainerDied","Data":"312c4a9994b8480c99ce8fc4529252e3518a48751d9011dc6b7784cb643fe8b0"} Nov 03 20:30:22.129146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.129123 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" event={"ID":"c9fce91d-067e-413a-8d5b-85d5b3d04e80","Type":"ContainerDied","Data":"ba851d17147bb729c36bc6dc92820dd4df4f63f9aaaf566faf8fc9606561f176"} Nov 03 20:30:22.129223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.129203 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm" Nov 03 20:30:22.136150 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.136124 4457 scope.go:122] "RemoveContainer" containerID="977a90dbe5af805764a35ef94986d7bd5bda8ea3a96fb8c2d3e393a4e9260714" Nov 03 20:30:22.139643 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.139618 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" event={"ID":"c1c5f7ba-2be4-4ca4-9223-7923c9ad9121","Type":"ContainerDied","Data":"8e8149882bf34bb56a9e10c57434288b8b1a03dc87e6d8e662a5430c9bea6a20"} Nov 03 20:30:22.139888 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.139869 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k" Nov 03 20:30:22.160019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.158172 4457 scope.go:122] "RemoveContainer" containerID="f2bfa1fb731f345cd8b4c9e1b31bf8eef11d83798e75ee5a2518181dd11cd2be" Nov 03 20:30:22.206473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.206421 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j"] Nov 03 20:30:22.216294 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.214753 4457 scope.go:122] "RemoveContainer" containerID="649829b918ffa9f52591b3db03cb8329a518de56693f8be534b32f19bb6b27b7" Nov 03 20:30:22.216294 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:30:22.215373 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1023951_0cef_4afe_b7d8_76d8a9775abb.slice/crio-df99e08ba2255ea4bde89daf0ccaadfe82d38dafe3d214dc5b56853f3b5bc5c6 WatchSource:0}: Error finding container df99e08ba2255ea4bde89daf0ccaadfe82d38dafe3d214dc5b56853f3b5bc5c6: Status 404 returned error can't find the container with id df99e08ba2255ea4bde89daf0ccaadfe82d38dafe3d214dc5b56853f3b5bc5c6 Nov 03 20:30:22.232349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.232273 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv"] Nov 03 20:30:22.249987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.249124 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-62221d-29895630-qm5nv"] Nov 03 20:30:22.254909 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.254119 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz"] Nov 03 20:30:22.336742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.336712 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm"] Nov 03 20:30:22.349208 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.349163 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-1c6a98-29895630-xn4cm"] Nov 03 20:30:22.365272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.365234 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k"] Nov 03 20:30:22.384100 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.383572 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-8013d9-29895630-4v55k"] Nov 03 20:30:22.392100 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.392054 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp"] Nov 03 20:30:22.398629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.398601 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-a744eb-29895630-6fstp"] Nov 03 20:30:22.453301 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.451564 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" Nov 03 20:30:22.558381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.558344 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-kube-api-access-9jgpf\" (UniqueName: \"kubernetes.io/projected/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-kube-api-access-9jgpf\") pod \"eea2c970-9b94-45c7-b8e7-e1263a11f8f1\" (UID: \"eea2c970-9b94-45c7-b8e7-e1263a11f8f1\") " Nov 03 20:30:22.558577 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.558556 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-maas-api-service-ca\") pod \"eea2c970-9b94-45c7-b8e7-e1263a11f8f1\" (UID: \"eea2c970-9b94-45c7-b8e7-e1263a11f8f1\") " Nov 03 20:30:22.558932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.558903 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-maas-api-service-ca" pod "eea2c970-9b94-45c7-b8e7-e1263a11f8f1" (UID: "eea2c970-9b94-45c7-b8e7-e1263a11f8f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:22.559634 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.559609 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.561116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.561073 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-kube-api-access-9jgpf" pod "eea2c970-9b94-45c7-b8e7-e1263a11f8f1" (UID: "eea2c970-9b94-45c7-b8e7-e1263a11f8f1"). InnerVolumeSpecName "kube-api-access-9jgpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:22.572076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.572053 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" Nov 03 20:30:22.575860 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.575840 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" Nov 03 20:30:22.666910 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.666825 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8d2d9a-946c-4e76-b319-c7d653ac2306-kube-api-access-4bx6r\" (UniqueName: \"kubernetes.io/projected/ce8d2d9a-946c-4e76-b319-c7d653ac2306-kube-api-access-4bx6r\") pod \"ce8d2d9a-946c-4e76-b319-c7d653ac2306\" (UID: \"ce8d2d9a-946c-4e76-b319-c7d653ac2306\") " Nov 03 20:30:22.668881 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.668856 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f52a0620-190b-418a-afdd-8c8ac053a1d4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52a0620-190b-418a-afdd-8c8ac053a1d4-maas-api-service-ca\") pod \"f52a0620-190b-418a-afdd-8c8ac053a1d4\" (UID: \"f52a0620-190b-418a-afdd-8c8ac053a1d4\") " Nov 03 20:30:22.669848 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.669827 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8d2d9a-946c-4e76-b319-c7d653ac2306-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce8d2d9a-946c-4e76-b319-c7d653ac2306-maas-api-service-ca\") pod \"ce8d2d9a-946c-4e76-b319-c7d653ac2306\" (UID: \"ce8d2d9a-946c-4e76-b319-c7d653ac2306\") " Nov 03 20:30:22.702785 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.669344 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f52a0620-190b-418a-afdd-8c8ac053a1d4-maas-api-service-ca" pod "f52a0620-190b-418a-afdd-8c8ac053a1d4" (UID: "f52a0620-190b-418a-afdd-8c8ac053a1d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:22.703004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.670332 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8d2d9a-946c-4e76-b319-c7d653ac2306-maas-api-service-ca" pod "ce8d2d9a-946c-4e76-b319-c7d653ac2306" (UID: "ce8d2d9a-946c-4e76-b319-c7d653ac2306"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:22.703125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.702953 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f52a0620-190b-418a-afdd-8c8ac053a1d4-kube-api-access-4s4mv\" (UniqueName: \"kubernetes.io/projected/f52a0620-190b-418a-afdd-8c8ac053a1d4-kube-api-access-4s4mv\") pod \"f52a0620-190b-418a-afdd-8c8ac053a1d4\" (UID: \"f52a0620-190b-418a-afdd-8c8ac053a1d4\") " Nov 03 20:30:22.706794 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.706766 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f52a0620-190b-418a-afdd-8c8ac053a1d4-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.706794 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.706794 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jgpf\" (UniqueName: \"kubernetes.io/projected/eea2c970-9b94-45c7-b8e7-e1263a11f8f1-kube-api-access-9jgpf\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.706995 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.706810 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce8d2d9a-946c-4e76-b319-c7d653ac2306-maas-api-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.707886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.707841 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8d2d9a-946c-4e76-b319-c7d653ac2306-kube-api-access-4bx6r" pod "ce8d2d9a-946c-4e76-b319-c7d653ac2306" (UID: "ce8d2d9a-946c-4e76-b319-c7d653ac2306"). InnerVolumeSpecName "kube-api-access-4bx6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:22.711160 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.711126 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52a0620-190b-418a-afdd-8c8ac053a1d4-kube-api-access-4s4mv" pod "f52a0620-190b-418a-afdd-8c8ac053a1d4" (UID: "f52a0620-190b-418a-afdd-8c8ac053a1d4"). InnerVolumeSpecName "kube-api-access-4s4mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:22.809237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.809212 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s4mv\" (UniqueName: \"kubernetes.io/projected/f52a0620-190b-418a-afdd-8c8ac053a1d4-kube-api-access-4s4mv\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:22.809441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:22.809422 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bx6r\" (UniqueName: \"kubernetes.io/projected/ce8d2d9a-946c-4e76-b319-c7d653ac2306-kube-api-access-4bx6r\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:23.162611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.162584 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" Nov 03 20:30:23.162611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.162594 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl" event={"ID":"eea2c970-9b94-45c7-b8e7-e1263a11f8f1","Type":"ContainerDied","Data":"a7e322a70c94ae145d2519f69c5786d4bc92a85d57e2d7035faa133419143733"} Nov 03 20:30:23.163112 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.162655 4457 scope.go:122] "RemoveContainer" containerID="e868d5de20cfe279fb210f9dfff00232f362181cfeae3e4585594da54aab9189" Nov 03 20:30:23.167216 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.167198 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" Nov 03 20:30:23.167366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.167241 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr" event={"ID":"f52a0620-190b-418a-afdd-8c8ac053a1d4","Type":"ContainerDied","Data":"d546ddc3e41d3b1d3dc1841df1d7257d3f5c6cb3f1df48646d77fe74599930ce"} Nov 03 20:30:23.170605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.170579 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" event={"ID":"ce8d2d9a-946c-4e76-b319-c7d653ac2306","Type":"ContainerDied","Data":"23da020ad82182ecd2f825912a4551e082e1190fe8f81cd8a78f5544db78a7af"} Nov 03 20:30:23.170729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.170674 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5" Nov 03 20:30:23.173796 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.173771 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" event={"ID":"b1023951-0cef-4afe-b7d8-76d8a9775abb","Type":"ContainerStarted","Data":"d4677b943b46a919d67bc54ac8bd765b502448fedc2c5f66d471f23938455081"} Nov 03 20:30:23.173915 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.173811 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" event={"ID":"b1023951-0cef-4afe-b7d8-76d8a9775abb","Type":"ContainerStarted","Data":"df99e08ba2255ea4bde89daf0ccaadfe82d38dafe3d214dc5b56853f3b5bc5c6"} Nov 03 20:30:23.176354 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.176328 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" event={"ID":"7f4de79d-18ac-4f57-803e-fb09ddcc8134","Type":"ContainerStarted","Data":"f499fcbd744a8309a84a3cf1f7cbb887981bce933155ed72a98928d40c5a011f"} Nov 03 20:30:23.176473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.176376 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" event={"ID":"7f4de79d-18ac-4f57-803e-fb09ddcc8134","Type":"ContainerStarted","Data":"e64a2e886dc73146e43970ceabe95fcccd23bde429606f7d1c97d6dce3bc906b"} Nov 03 20:30:23.178553 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.178529 4457 scope.go:122] "RemoveContainer" containerID="b9ac36d9b5acca790b02cd5fcb286bbe3b28d172f4ed958685c0cc49ef902886" Nov 03 20:30:23.194519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.194494 4457 scope.go:122] "RemoveContainer" containerID="b376975b35a90c605c1420cdee8d4a373bf1c227c3e3fe03fb0cdf228206a8f2" Nov 03 20:30:23.195264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.195235 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl"] Nov 03 20:30:23.200770 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.200746 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-90ef9c-29895630-9nhrl"] Nov 03 20:30:23.214743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.214689 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" podStartSLOduration=2.214672438 podStartE2EDuration="2.214672438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:23.210709232 +0000 UTC m=+2001.586305979" watchObservedRunningTime="2026-11-03 20:30:23.214672438 +0000 UTC m=+2001.590269187" Nov 03 20:30:23.231221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.231175 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr"] Nov 03 20:30:23.237459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.237437 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-45e569-29895630-x86qr"] Nov 03 20:30:23.250232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.250180 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" podStartSLOduration=2.25016455 podStartE2EDuration="2.25016455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:30:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:30:23.249118455 +0000 UTC m=+2001.624715202" watchObservedRunningTime="2026-11-03 20:30:23.25016455 +0000 UTC m=+2001.625761298" Nov 03 20:30:23.271928 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.271900 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5"] Nov 03 20:30:23.279593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.279567 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5ca474-29895630-wv4l5"] Nov 03 20:30:23.764984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.764951 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cad965b-7c57-4560-921e-0e5ed244784a" path="/var/lib/kubelet/pods/6cad965b-7c57-4560-921e-0e5ed244784a/volumes" Nov 03 20:30:23.765524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.765508 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" path="/var/lib/kubelet/pods/c1c5f7ba-2be4-4ca4-9223-7923c9ad9121/volumes" Nov 03 20:30:23.765968 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.765952 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" path="/var/lib/kubelet/pods/c9fce91d-067e-413a-8d5b-85d5b3d04e80/volumes" Nov 03 20:30:23.766769 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.766750 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" path="/var/lib/kubelet/pods/ce8d2d9a-946c-4e76-b319-c7d653ac2306/volumes" Nov 03 20:30:23.767179 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.767164 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d168551a-ced0-4313-83f4-6f1d4f3b758d" path="/var/lib/kubelet/pods/d168551a-ced0-4313-83f4-6f1d4f3b758d/volumes" Nov 03 20:30:23.767617 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.767601 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" path="/var/lib/kubelet/pods/eea2c970-9b94-45c7-b8e7-e1263a11f8f1/volumes" Nov 03 20:30:23.768291 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.768276 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f52a0620-190b-418a-afdd-8c8ac053a1d4" path="/var/lib/kubelet/pods/f52a0620-190b-418a-afdd-8c8ac053a1d4/volumes" Nov 03 20:30:23.828223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:23.828131 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" podUID="3bc0361e-b362-496f-8d29-ac11c012caa6" containerName="payload-pre-processing" containerID="cri-o://08cc367d3a242cee5e47df3ee9479bb4de4687409b6d7132527e4061874dbd95" gracePeriod=25 Nov 03 20:30:24.193711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:24.193618 4457 generic.go:356] "Generic (PLEG): container finished" podID="b1023951-0cef-4afe-b7d8-76d8a9775abb" containerID="d4677b943b46a919d67bc54ac8bd765b502448fedc2c5f66d471f23938455081" exitCode=0 Nov 03 20:30:24.194186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:24.193709 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" event={"ID":"b1023951-0cef-4afe-b7d8-76d8a9775abb","Type":"ContainerDied","Data":"d4677b943b46a919d67bc54ac8bd765b502448fedc2c5f66d471f23938455081"} Nov 03 20:30:24.195586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:24.195562 4457 generic.go:356] "Generic (PLEG): container finished" podID="7f4de79d-18ac-4f57-803e-fb09ddcc8134" containerID="f499fcbd744a8309a84a3cf1f7cbb887981bce933155ed72a98928d40c5a011f" exitCode=0 Nov 03 20:30:24.195690 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:24.195642 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" event={"ID":"7f4de79d-18ac-4f57-803e-fb09ddcc8134","Type":"ContainerDied","Data":"f499fcbd744a8309a84a3cf1f7cbb887981bce933155ed72a98928d40c5a011f"} Nov 03 20:30:25.434371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.434340 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" Nov 03 20:30:25.471560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.471527 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1023951-0cef-4afe-b7d8-76d8a9775abb-kube-api-access-2tf2t\" (UniqueName: \"kubernetes.io/projected/b1023951-0cef-4afe-b7d8-76d8a9775abb-kube-api-access-2tf2t\") pod \"b1023951-0cef-4afe-b7d8-76d8a9775abb\" (UID: \"b1023951-0cef-4afe-b7d8-76d8a9775abb\") " Nov 03 20:30:25.471803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.471781 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-openshift-ca\") pod \"b1023951-0cef-4afe-b7d8-76d8a9775abb\" (UID: \"b1023951-0cef-4afe-b7d8-76d8a9775abb\") " Nov 03 20:30:25.472319 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.472291 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-xks-ca\") pod \"b1023951-0cef-4afe-b7d8-76d8a9775abb\" (UID: \"b1023951-0cef-4afe-b7d8-76d8a9775abb\") " Nov 03 20:30:25.474373 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.474331 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-openshift-ca" pod "b1023951-0cef-4afe-b7d8-76d8a9775abb" (UID: "b1023951-0cef-4afe-b7d8-76d8a9775abb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:25.480850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.480819 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-xks-ca" pod "b1023951-0cef-4afe-b7d8-76d8a9775abb" (UID: "b1023951-0cef-4afe-b7d8-76d8a9775abb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:25.482521 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.482495 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1023951-0cef-4afe-b7d8-76d8a9775abb-kube-api-access-2tf2t" pod "b1023951-0cef-4afe-b7d8-76d8a9775abb" (UID: "b1023951-0cef-4afe-b7d8-76d8a9775abb"). InnerVolumeSpecName "kube-api-access-2tf2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:25.529345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.529312 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" Nov 03 20:30:25.574804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.574633 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-xks-ca\") pod \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\" (UID: \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\") " Nov 03 20:30:25.574976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.574888 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4de79d-18ac-4f57-803e-fb09ddcc8134-kube-api-access-jzdrd\" (UniqueName: \"kubernetes.io/projected/7f4de79d-18ac-4f57-803e-fb09ddcc8134-kube-api-access-jzdrd\") pod \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\" (UID: \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\") " Nov 03 20:30:25.575080 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.575058 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-openshift-ca\") pod \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\" (UID: \"7f4de79d-18ac-4f57-803e-fb09ddcc8134\") " Nov 03 20:30:25.575849 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.575821 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:25.576034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.575854 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tf2t\" (UniqueName: \"kubernetes.io/projected/b1023951-0cef-4afe-b7d8-76d8a9775abb-kube-api-access-2tf2t\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:25.576034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.575871 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1023951-0cef-4afe-b7d8-76d8a9775abb-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:25.589975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.589941 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4de79d-18ac-4f57-803e-fb09ddcc8134-kube-api-access-jzdrd" pod "7f4de79d-18ac-4f57-803e-fb09ddcc8134" (UID: "7f4de79d-18ac-4f57-803e-fb09ddcc8134"). InnerVolumeSpecName "kube-api-access-jzdrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:25.591109 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.591081 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-openshift-ca" pod "7f4de79d-18ac-4f57-803e-fb09ddcc8134" (UID: "7f4de79d-18ac-4f57-803e-fb09ddcc8134"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:25.591540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.591511 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-xks-ca" pod "7f4de79d-18ac-4f57-803e-fb09ddcc8134" (UID: "7f4de79d-18ac-4f57-803e-fb09ddcc8134"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:25.677875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.677824 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzdrd\" (UniqueName: \"kubernetes.io/projected/7f4de79d-18ac-4f57-803e-fb09ddcc8134-kube-api-access-jzdrd\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:25.677875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.677873 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:25.678133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:25.677892 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f4de79d-18ac-4f57-803e-fb09ddcc8134-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:26.223670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:26.223637 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" Nov 03 20:30:26.223858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:26.223669 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j" event={"ID":"b1023951-0cef-4afe-b7d8-76d8a9775abb","Type":"ContainerDied","Data":"df99e08ba2255ea4bde89daf0ccaadfe82d38dafe3d214dc5b56853f3b5bc5c6"} Nov 03 20:30:26.223858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:26.223720 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df99e08ba2255ea4bde89daf0ccaadfe82d38dafe3d214dc5b56853f3b5bc5c6" Nov 03 20:30:26.225302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:26.225264 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" event={"ID":"7f4de79d-18ac-4f57-803e-fb09ddcc8134","Type":"ContainerDied","Data":"e64a2e886dc73146e43970ceabe95fcccd23bde429606f7d1c97d6dce3bc906b"} Nov 03 20:30:26.225523 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:26.225321 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e64a2e886dc73146e43970ceabe95fcccd23bde429606f7d1c97d6dce3bc906b" Nov 03 20:30:26.225523 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:26.225275 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz" Nov 03 20:30:28.362517 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.362465 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" containerID="cri-o://07cd3f211bbbce3c253aba8d4fe8d2644239cc11dbd553da41a9dc3cb91d740c" gracePeriod=45 Nov 03 20:30:28.401974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.401939 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" containerID="cri-o://69a70f88ecb774f0acc80679a1fc73a3e9b5e9236ca635484edece286823ecff" gracePeriod=45 Nov 03 20:30:28.807301 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.806527 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:30:28.865744 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.865709 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:30:28.892057 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.892021 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-dshm\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-dshm\") pod \"555064e0-9ca2-47de-b1c0-8473f3729219\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " Nov 03 20:30:28.892233 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.892095 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6387668-070a-49e7-ab69-62ca763db030-kube-api-access-8jkc7\" (UniqueName: \"kubernetes.io/projected/e6387668-070a-49e7-ab69-62ca763db030-kube-api-access-8jkc7\") pod \"e6387668-070a-49e7-ab69-62ca763db030\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " Nov 03 20:30:28.892233 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.892222 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-model-cache\") pod \"e6387668-070a-49e7-ab69-62ca763db030\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " Nov 03 20:30:28.892341 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.892256 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-home\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-home\") pod \"e6387668-070a-49e7-ab69-62ca763db030\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " Nov 03 20:30:28.892341 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.892287 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/555064e0-9ca2-47de-b1c0-8473f3729219-tls-certs\" (UniqueName: \"kubernetes.io/secret/555064e0-9ca2-47de-b1c0-8473f3729219-tls-certs\") pod \"555064e0-9ca2-47de-b1c0-8473f3729219\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " Nov 03 20:30:28.892341 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.892329 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-model-cache\") pod \"555064e0-9ca2-47de-b1c0-8473f3729219\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " Nov 03 20:30:28.892581 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.892459 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-tmp-dir\") pod \"e6387668-070a-49e7-ab69-62ca763db030\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " Nov 03 20:30:28.892581 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.892490 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-home\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-home\") pod \"555064e0-9ca2-47de-b1c0-8473f3729219\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " Nov 03 20:30:28.892581 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.892564 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-dshm\") pod \"e6387668-070a-49e7-ab69-62ca763db030\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " Nov 03 20:30:28.892734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.892602 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6387668-070a-49e7-ab69-62ca763db030-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6387668-070a-49e7-ab69-62ca763db030-tls-certs\") pod \"e6387668-070a-49e7-ab69-62ca763db030\" (UID: \"e6387668-070a-49e7-ab69-62ca763db030\") " Nov 03 20:30:28.892734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.892644 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/555064e0-9ca2-47de-b1c0-8473f3729219-kube-api-access-dq94q\" (UniqueName: \"kubernetes.io/projected/555064e0-9ca2-47de-b1c0-8473f3729219-kube-api-access-dq94q\") pod \"555064e0-9ca2-47de-b1c0-8473f3729219\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " Nov 03 20:30:28.892734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.892704 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-tmp-dir\") pod \"555064e0-9ca2-47de-b1c0-8473f3729219\" (UID: \"555064e0-9ca2-47de-b1c0-8473f3729219\") " Nov 03 20:30:28.893530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.893499 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-model-cache" pod "555064e0-9ca2-47de-b1c0-8473f3729219" (UID: "555064e0-9ca2-47de-b1c0-8473f3729219"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.893725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.893660 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-model-cache" pod "e6387668-070a-49e7-ab69-62ca763db030" (UID: "e6387668-070a-49e7-ab69-62ca763db030"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.893961 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.893934 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-tmp-dir" pod "555064e0-9ca2-47de-b1c0-8473f3729219" (UID: "555064e0-9ca2-47de-b1c0-8473f3729219"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.894147 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.894125 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-home" pod "e6387668-070a-49e7-ab69-62ca763db030" (UID: "e6387668-070a-49e7-ab69-62ca763db030"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.894448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.894428 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-home" pod "555064e0-9ca2-47de-b1c0-8473f3729219" (UID: "555064e0-9ca2-47de-b1c0-8473f3729219"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.894836 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.894809 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-tmp-dir" pod "e6387668-070a-49e7-ab69-62ca763db030" (UID: "e6387668-070a-49e7-ab69-62ca763db030"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.897057 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.897034 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6387668-070a-49e7-ab69-62ca763db030-kube-api-access-8jkc7" pod "e6387668-070a-49e7-ab69-62ca763db030" (UID: "e6387668-070a-49e7-ab69-62ca763db030"). InnerVolumeSpecName "kube-api-access-8jkc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:28.897495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.897467 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/555064e0-9ca2-47de-b1c0-8473f3729219-kube-api-access-dq94q" pod "555064e0-9ca2-47de-b1c0-8473f3729219" (UID: "555064e0-9ca2-47de-b1c0-8473f3729219"). InnerVolumeSpecName "kube-api-access-dq94q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:28.898159 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.898121 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/555064e0-9ca2-47de-b1c0-8473f3729219-tls-certs" pod "555064e0-9ca2-47de-b1c0-8473f3729219" (UID: "555064e0-9ca2-47de-b1c0-8473f3729219"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:28.899732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.899690 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-dshm" pod "e6387668-070a-49e7-ab69-62ca763db030" (UID: "e6387668-070a-49e7-ab69-62ca763db030"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.899965 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.899943 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-dshm" pod "555064e0-9ca2-47de-b1c0-8473f3729219" (UID: "555064e0-9ca2-47de-b1c0-8473f3729219"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:28.900229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:28.900199 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6387668-070a-49e7-ab69-62ca763db030-tls-certs" pod "e6387668-070a-49e7-ab69-62ca763db030" (UID: "e6387668-070a-49e7-ab69-62ca763db030"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:30:29.002530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.001249 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:29.002530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.001291 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:29.002530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.001309 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jkc7\" (UniqueName: \"kubernetes.io/projected/e6387668-070a-49e7-ab69-62ca763db030-kube-api-access-8jkc7\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:29.002530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.001326 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:29.002530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.001340 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:29.002530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.001353 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/555064e0-9ca2-47de-b1c0-8473f3729219-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:29.002530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.001365 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:29.002530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.001380 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:29.002530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.001411 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/555064e0-9ca2-47de-b1c0-8473f3729219-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:29.002530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.001425 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6387668-070a-49e7-ab69-62ca763db030-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:29.002530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.001437 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6387668-070a-49e7-ab69-62ca763db030-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:29.002530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.001450 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq94q\" (UniqueName: \"kubernetes.io/projected/555064e0-9ca2-47de-b1c0-8473f3729219-kube-api-access-dq94q\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:29.273181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.273093 4457 generic.go:356] "Generic (PLEG): container finished" podID="555064e0-9ca2-47de-b1c0-8473f3729219" containerID="69a70f88ecb774f0acc80679a1fc73a3e9b5e9236ca635484edece286823ecff" exitCode=0 Nov 03 20:30:29.273181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.273151 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" Nov 03 20:30:29.273438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.273202 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" event={"ID":"555064e0-9ca2-47de-b1c0-8473f3729219","Type":"ContainerDied","Data":"69a70f88ecb774f0acc80679a1fc73a3e9b5e9236ca635484edece286823ecff"} Nov 03 20:30:29.273438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.273248 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" event={"ID":"555064e0-9ca2-47de-b1c0-8473f3729219","Type":"ContainerDied","Data":"0d74dc6b0d869b3fb3d1b9a0bab939395b24a8f08101eb4c5736797a702c7a04"} Nov 03 20:30:29.273438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.273282 4457 scope.go:122] "RemoveContainer" containerID="69a70f88ecb774f0acc80679a1fc73a3e9b5e9236ca635484edece286823ecff" Nov 03 20:30:29.275528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.275502 4457 generic.go:356] "Generic (PLEG): container finished" podID="e6387668-070a-49e7-ab69-62ca763db030" containerID="07cd3f211bbbce3c253aba8d4fe8d2644239cc11dbd553da41a9dc3cb91d740c" exitCode=0 Nov 03 20:30:29.275635 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.275548 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" Nov 03 20:30:29.275819 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.275794 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" event={"ID":"e6387668-070a-49e7-ab69-62ca763db030","Type":"ContainerDied","Data":"07cd3f211bbbce3c253aba8d4fe8d2644239cc11dbd553da41a9dc3cb91d740c"} Nov 03 20:30:29.275875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.275834 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95" event={"ID":"e6387668-070a-49e7-ab69-62ca763db030","Type":"ContainerDied","Data":"b3c7cb79990d4714292a567688d89276c8a75d8a157779881f4c3106e5d094d8"} Nov 03 20:30:29.298246 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.298178 4457 scope.go:122] "RemoveContainer" containerID="69a70f88ecb774f0acc80679a1fc73a3e9b5e9236ca635484edece286823ecff" Nov 03 20:30:29.298945 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:29.298911 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a70f88ecb774f0acc80679a1fc73a3e9b5e9236ca635484edece286823ecff\": container with ID starting with 69a70f88ecb774f0acc80679a1fc73a3e9b5e9236ca635484edece286823ecff not found: ID does not exist" containerID="69a70f88ecb774f0acc80679a1fc73a3e9b5e9236ca635484edece286823ecff" Nov 03 20:30:29.299052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.298952 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a70f88ecb774f0acc80679a1fc73a3e9b5e9236ca635484edece286823ecff"} err="failed to get container status \"69a70f88ecb774f0acc80679a1fc73a3e9b5e9236ca635484edece286823ecff\": rpc error: code = NotFound desc = could not find container \"69a70f88ecb774f0acc80679a1fc73a3e9b5e9236ca635484edece286823ecff\": container with ID starting with 69a70f88ecb774f0acc80679a1fc73a3e9b5e9236ca635484edece286823ecff not found: ID does not exist" Nov 03 20:30:29.299052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.298975 4457 scope.go:122] "RemoveContainer" containerID="07cd3f211bbbce3c253aba8d4fe8d2644239cc11dbd553da41a9dc3cb91d740c" Nov 03 20:30:29.311522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.311500 4457 scope.go:122] "RemoveContainer" containerID="07cd3f211bbbce3c253aba8d4fe8d2644239cc11dbd553da41a9dc3cb91d740c" Nov 03 20:30:29.311832 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:29.311811 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07cd3f211bbbce3c253aba8d4fe8d2644239cc11dbd553da41a9dc3cb91d740c\": container with ID starting with 07cd3f211bbbce3c253aba8d4fe8d2644239cc11dbd553da41a9dc3cb91d740c not found: ID does not exist" containerID="07cd3f211bbbce3c253aba8d4fe8d2644239cc11dbd553da41a9dc3cb91d740c" Nov 03 20:30:29.311878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.311839 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07cd3f211bbbce3c253aba8d4fe8d2644239cc11dbd553da41a9dc3cb91d740c"} err="failed to get container status \"07cd3f211bbbce3c253aba8d4fe8d2644239cc11dbd553da41a9dc3cb91d740c\": rpc error: code = NotFound desc = could not find container \"07cd3f211bbbce3c253aba8d4fe8d2644239cc11dbd553da41a9dc3cb91d740c\": container with ID starting with 07cd3f211bbbce3c253aba8d4fe8d2644239cc11dbd553da41a9dc3cb91d740c not found: ID does not exist" Nov 03 20:30:29.315462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.315438 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r"] Nov 03 20:30:29.320612 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.320592 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r"] Nov 03 20:30:29.334630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.334158 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95"] Nov 03 20:30:29.339595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.339570 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8013d9/distinct-2-w1-8013d9-kserve-5946cb8b8-qgl95"] Nov 03 20:30:29.763202 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.763161 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" path="/var/lib/kubelet/pods/555064e0-9ca2-47de-b1c0-8473f3729219/volumes" Nov 03 20:30:29.763810 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.763793 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6387668-070a-49e7-ab69-62ca763db030" path="/var/lib/kubelet/pods/e6387668-070a-49e7-ab69-62ca763db030/volumes" Nov 03 20:30:29.771360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:29.771332 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8013d9/simulator-w1-8013d9-kserve-7d969b758f-kmh4r" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/ready\": context deadline exceeded" Nov 03 20:30:32.970044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:32.970000 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" podUID="3bc0361e-b362-496f-8d29-ac11c012caa6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:30:33.028448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:33.028411 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656"] Nov 03 20:30:33.036451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:33.036423 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461f444c-pz656"] Nov 03 20:30:33.762611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:33.762578 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031f50c1-6a58-454b-a132-2291aa316cd3" path="/var/lib/kubelet/pods/031f50c1-6a58-454b-a132-2291aa316cd3/volumes" Nov 03 20:30:36.147971 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:36.147936 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:30:42.970558 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:42.970507 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" podUID="3bc0361e-b362-496f-8d29-ac11c012caa6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:30:44.148526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.148499 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" Nov 03 20:30:44.205700 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.205654 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc0361e-b362-496f-8d29-ac11c012caa6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc0361e-b362-496f-8d29-ac11c012caa6-plugins-config-volume\") pod \"3bc0361e-b362-496f-8d29-ac11c012caa6\" (UID: \"3bc0361e-b362-496f-8d29-ac11c012caa6\") " Nov 03 20:30:44.206188 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.206167 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc0361e-b362-496f-8d29-ac11c012caa6-kube-api-access-qtfkm\" (UniqueName: \"kubernetes.io/projected/3bc0361e-b362-496f-8d29-ac11c012caa6-kube-api-access-qtfkm\") pod \"3bc0361e-b362-496f-8d29-ac11c012caa6\" (UID: \"3bc0361e-b362-496f-8d29-ac11c012caa6\") " Nov 03 20:30:44.207286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.207255 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc0361e-b362-496f-8d29-ac11c012caa6-plugins-config-volume" pod "3bc0361e-b362-496f-8d29-ac11c012caa6" (UID: "3bc0361e-b362-496f-8d29-ac11c012caa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:44.213218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.210527 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc0361e-b362-496f-8d29-ac11c012caa6-kube-api-access-qtfkm" pod "3bc0361e-b362-496f-8d29-ac11c012caa6" (UID: "3bc0361e-b362-496f-8d29-ac11c012caa6"). InnerVolumeSpecName "kube-api-access-qtfkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:44.309182 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.309144 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtfkm\" (UniqueName: \"kubernetes.io/projected/3bc0361e-b362-496f-8d29-ac11c012caa6-kube-api-access-qtfkm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:44.309182 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.309177 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bc0361e-b362-496f-8d29-ac11c012caa6-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:44.477724 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.477554 4457 generic.go:356] "Generic (PLEG): container finished" podID="3bc0361e-b362-496f-8d29-ac11c012caa6" containerID="08cc367d3a242cee5e47df3ee9479bb4de4687409b6d7132527e4061874dbd95" exitCode=0 Nov 03 20:30:44.478081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.477603 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" Nov 03 20:30:44.478229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.477615 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" event={"ID":"3bc0361e-b362-496f-8d29-ac11c012caa6","Type":"ContainerDied","Data":"08cc367d3a242cee5e47df3ee9479bb4de4687409b6d7132527e4061874dbd95"} Nov 03 20:30:44.478543 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.478518 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg" event={"ID":"3bc0361e-b362-496f-8d29-ac11c012caa6","Type":"ContainerDied","Data":"06a1d5ad61c6892ca0c1a3a4e4bb376881d9afb2403bf2de7901db963b5f1591"} Nov 03 20:30:44.478715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.478698 4457 scope.go:122] "RemoveContainer" containerID="08cc367d3a242cee5e47df3ee9479bb4de4687409b6d7132527e4061874dbd95" Nov 03 20:30:44.500516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.500489 4457 scope.go:122] "RemoveContainer" containerID="08cc367d3a242cee5e47df3ee9479bb4de4687409b6d7132527e4061874dbd95" Nov 03 20:30:44.500910 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:44.500882 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08cc367d3a242cee5e47df3ee9479bb4de4687409b6d7132527e4061874dbd95\": container with ID starting with 08cc367d3a242cee5e47df3ee9479bb4de4687409b6d7132527e4061874dbd95 not found: ID does not exist" containerID="08cc367d3a242cee5e47df3ee9479bb4de4687409b6d7132527e4061874dbd95" Nov 03 20:30:44.501000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.500922 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08cc367d3a242cee5e47df3ee9479bb4de4687409b6d7132527e4061874dbd95"} err="failed to get container status \"08cc367d3a242cee5e47df3ee9479bb4de4687409b6d7132527e4061874dbd95\": rpc error: code = NotFound desc = could not find container \"08cc367d3a242cee5e47df3ee9479bb4de4687409b6d7132527e4061874dbd95\": container with ID starting with 08cc367d3a242cee5e47df3ee9479bb4de4687409b6d7132527e4061874dbd95 not found: ID does not exist" Nov 03 20:30:44.514173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.514137 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg"] Nov 03 20:30:44.520170 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:44.520137 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-8013d9-78dc8df65f-xcdxg"] Nov 03 20:30:45.768412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:45.767796 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc0361e-b362-496f-8d29-ac11c012caa6" path="/var/lib/kubelet/pods/3bc0361e-b362-496f-8d29-ac11c012caa6/volumes" Nov 03 20:30:48.906854 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:48.906763 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh"] Nov 03 20:30:48.908000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:48.907808 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" containerID="cri-o://8be577314845fdde68c2420c09a95b2868a2f30cfe04215a565f30523bc146ad" gracePeriod=30 Nov 03 20:30:53.997415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:53.995197 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:30:54.112863 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.075754 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 03 20:30:54.208431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.207435 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:30:54.254535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.254475 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh"] Nov 03 20:30:54.255155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.255121 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerName="istio-proxy" containerID="cri-o://e16da9771721620bbd31c0ecda2c81095f716e4ff819726fd8f57416105118db" gracePeriod=30 Nov 03 20:30:54.277563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.277513 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" podUID="2699c9a3-baf0-40f0-b1be-892b29a62ad4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.194:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:30:54.368572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.353550 4457 generic.go:356] "Generic (PLEG): container finished" podID="62c925c7-63db-46a9-a936-a0a56f28e396" containerID="8be577314845fdde68c2420c09a95b2868a2f30cfe04215a565f30523bc146ad" exitCode=0 Nov 03 20:30:54.368572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.353802 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" event={"ID":"62c925c7-63db-46a9-a936-a0a56f28e396","Type":"ContainerDied","Data":"8be577314845fdde68c2420c09a95b2868a2f30cfe04215a565f30523bc146ad"} Nov 03 20:30:54.436413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.422601 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" podUID="75a384ec-5e7f-4f72-9381-92b564160ef6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.198:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:30:54.627381 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:30:54.627341 4457 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c925c7_63db_46a9_a936_a0a56f28e396.slice/crio-8be577314845fdde68c2420c09a95b2868a2f30cfe04215a565f30523bc146ad.scope\": RecentStats: unable to find data in memory cache]" Nov 03 20:30:54.808441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.808366 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:30:54.835287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.835250 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c925c7-63db-46a9-a936-a0a56f28e396-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c925c7-63db-46a9-a936-a0a56f28e396-istiod-ca-cert\") pod \"62c925c7-63db-46a9-a936-a0a56f28e396\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " Nov 03 20:30:54.835517 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.835308 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-socket\") pod \"62c925c7-63db-46a9-a936-a0a56f28e396\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " Nov 03 20:30:54.835517 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.835328 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-data\") pod \"62c925c7-63db-46a9-a936-a0a56f28e396\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " Nov 03 20:30:54.835829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.835711 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c925c7-63db-46a9-a936-a0a56f28e396-istiod-ca-cert" pod "62c925c7-63db-46a9-a936-a0a56f28e396" (UID: "62c925c7-63db-46a9-a936-a0a56f28e396"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:30:54.835829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.835794 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-data" pod "62c925c7-63db-46a9-a936-a0a56f28e396" (UID: "62c925c7-63db-46a9-a936-a0a56f28e396"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.835829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.835728 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-socket" pod "62c925c7-63db-46a9-a936-a0a56f28e396" (UID: "62c925c7-63db-46a9-a936-a0a56f28e396"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.836039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.835969 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c925c7-63db-46a9-a936-a0a56f28e396-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.836039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.835991 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.836039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.836007 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.937034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.936995 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-certs\") pod \"62c925c7-63db-46a9-a936-a0a56f28e396\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " Nov 03 20:30:54.937238 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.937099 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-kube-api-access-24n8w\" (UniqueName: \"kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-kube-api-access-24n8w\") pod \"62c925c7-63db-46a9-a936-a0a56f28e396\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " Nov 03 20:30:54.938172 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.937299 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-credential-socket\") pod \"62c925c7-63db-46a9-a936-a0a56f28e396\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " Nov 03 20:30:54.939103 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.939080 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62c925c7-63db-46a9-a936-a0a56f28e396-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c925c7-63db-46a9-a936-a0a56f28e396-istio-podinfo\") pod \"62c925c7-63db-46a9-a936-a0a56f28e396\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " Nov 03 20:30:54.939226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.937828 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-certs" pod "62c925c7-63db-46a9-a936-a0a56f28e396" (UID: "62c925c7-63db-46a9-a936-a0a56f28e396"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.939226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.939023 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-credential-socket" pod "62c925c7-63db-46a9-a936-a0a56f28e396" (UID: "62c925c7-63db-46a9-a936-a0a56f28e396"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.939226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.939219 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-envoy\") pod \"62c925c7-63db-46a9-a936-a0a56f28e396\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " Nov 03 20:30:54.939447 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.939257 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-istio-token\" (UniqueName: \"kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-istio-token\") pod \"62c925c7-63db-46a9-a936-a0a56f28e396\" (UID: \"62c925c7-63db-46a9-a936-a0a56f28e396\") " Nov 03 20:30:54.940265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.940242 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.940753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.940732 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:54.940890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.940834 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-kube-api-access-24n8w" pod "62c925c7-63db-46a9-a936-a0a56f28e396" (UID: "62c925c7-63db-46a9-a936-a0a56f28e396"). InnerVolumeSpecName "kube-api-access-24n8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:54.943944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.943921 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-envoy" pod "62c925c7-63db-46a9-a936-a0a56f28e396" (UID: "62c925c7-63db-46a9-a936-a0a56f28e396"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:30:54.948186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.948162 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62c925c7-63db-46a9-a936-a0a56f28e396-istio-podinfo" pod "62c925c7-63db-46a9-a936-a0a56f28e396" (UID: "62c925c7-63db-46a9-a936-a0a56f28e396"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:30:54.954328 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:54.954287 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-istio-token" pod "62c925c7-63db-46a9-a936-a0a56f28e396" (UID: "62c925c7-63db-46a9-a936-a0a56f28e396"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:30:55.042282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:55.042200 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c925c7-63db-46a9-a936-a0a56f28e396-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.042282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:55.042233 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c925c7-63db-46a9-a936-a0a56f28e396-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.042282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:55.042242 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.042282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:55.042251 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24n8w\" (UniqueName: \"kubernetes.io/projected/62c925c7-63db-46a9-a936-a0a56f28e396-kube-api-access-24n8w\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:30:55.367432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:55.367383 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" Nov 03 20:30:55.367627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:55.367447 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh" event={"ID":"62c925c7-63db-46a9-a936-a0a56f28e396","Type":"ContainerDied","Data":"529bf57a082b5d21e1fa12d89dc794bafe7fd8b54727e8ccb8e8a94755972105"} Nov 03 20:30:55.367627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:55.367522 4457 scope.go:122] "RemoveContainer" containerID="8be577314845fdde68c2420c09a95b2868a2f30cfe04215a565f30523bc146ad" Nov 03 20:30:55.455532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:55.455469 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh"] Nov 03 20:30:55.464344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:55.464308 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-bn7nh"] Nov 03 20:30:55.764926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:55.764845 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" path="/var/lib/kubelet/pods/62c925c7-63db-46a9-a936-a0a56f28e396/volumes" Nov 03 20:30:56.632117 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:56.632079 4457 patch_prober.go:29] interesting pod/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:30:56.632658 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:30:56.632136 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:31:00.221960 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.221926 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:31:00.369769 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.369730 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-certs\") pod \"ab0fbd33-7b13-4702-b80d-a102f28187be\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " Nov 03 20:31:00.369971 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.369783 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-data\") pod \"ab0fbd33-7b13-4702-b80d-a102f28187be\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " Nov 03 20:31:00.369971 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.369888 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-podinfo\") pod \"ab0fbd33-7b13-4702-b80d-a102f28187be\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " Nov 03 20:31:00.369971 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.369933 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-envoy\") pod \"ab0fbd33-7b13-4702-b80d-a102f28187be\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " Nov 03 20:31:00.370162 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.370002 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-socket\") pod \"ab0fbd33-7b13-4702-b80d-a102f28187be\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " Nov 03 20:31:00.370162 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.370070 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-token\" (UniqueName: \"kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-token\") pod \"ab0fbd33-7b13-4702-b80d-a102f28187be\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " Nov 03 20:31:00.370162 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.370076 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-certs" pod "ab0fbd33-7b13-4702-b80d-a102f28187be" (UID: "ab0fbd33-7b13-4702-b80d-a102f28187be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.370295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.370165 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-credential-socket\") pod \"ab0fbd33-7b13-4702-b80d-a102f28187be\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " Nov 03 20:31:00.370295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.370250 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab0fbd33-7b13-4702-b80d-a102f28187be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0fbd33-7b13-4702-b80d-a102f28187be-istiod-ca-cert\") pod \"ab0fbd33-7b13-4702-b80d-a102f28187be\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " Nov 03 20:31:00.370459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.370369 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-kube-api-access-zn2fp\" (UniqueName: \"kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-kube-api-access-zn2fp\") pod \"ab0fbd33-7b13-4702-b80d-a102f28187be\" (UID: \"ab0fbd33-7b13-4702-b80d-a102f28187be\") " Nov 03 20:31:00.370597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.370570 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-credential-socket" pod "ab0fbd33-7b13-4702-b80d-a102f28187be" (UID: "ab0fbd33-7b13-4702-b80d-a102f28187be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.371353 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.371323 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-socket" pod "ab0fbd33-7b13-4702-b80d-a102f28187be" (UID: "ab0fbd33-7b13-4702-b80d-a102f28187be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.371515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.371400 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.371515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.371425 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.371515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.371502 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0fbd33-7b13-4702-b80d-a102f28187be-istiod-ca-cert" pod "ab0fbd33-7b13-4702-b80d-a102f28187be" (UID: "ab0fbd33-7b13-4702-b80d-a102f28187be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:00.371738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.371714 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-data" pod "ab0fbd33-7b13-4702-b80d-a102f28187be" (UID: "ab0fbd33-7b13-4702-b80d-a102f28187be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.373936 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.373908 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-podinfo" pod "ab0fbd33-7b13-4702-b80d-a102f28187be" (UID: "ab0fbd33-7b13-4702-b80d-a102f28187be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:31:00.380586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.380555 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-token" pod "ab0fbd33-7b13-4702-b80d-a102f28187be" (UID: "ab0fbd33-7b13-4702-b80d-a102f28187be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:00.380970 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.380922 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-envoy" pod "ab0fbd33-7b13-4702-b80d-a102f28187be" (UID: "ab0fbd33-7b13-4702-b80d-a102f28187be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:00.381645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.381601 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-kube-api-access-zn2fp" pod "ab0fbd33-7b13-4702-b80d-a102f28187be" (UID: "ab0fbd33-7b13-4702-b80d-a102f28187be"). InnerVolumeSpecName "kube-api-access-zn2fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:00.429132 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.429102 4457 generic.go:356] "Generic (PLEG): container finished" podID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerID="e16da9771721620bbd31c0ecda2c81095f716e4ff819726fd8f57416105118db" exitCode=0 Nov 03 20:31:00.429268 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.429153 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" Nov 03 20:31:00.429268 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.429186 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" event={"ID":"ab0fbd33-7b13-4702-b80d-a102f28187be","Type":"ContainerDied","Data":"e16da9771721620bbd31c0ecda2c81095f716e4ff819726fd8f57416105118db"} Nov 03 20:31:00.429268 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.429241 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh" event={"ID":"ab0fbd33-7b13-4702-b80d-a102f28187be","Type":"ContainerDied","Data":"d9dfd87305379ac114d625de4fb8cb03cfc0b976e4d9cce6d27045e331e40000"} Nov 03 20:31:00.429439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.429269 4457 scope.go:122] "RemoveContainer" containerID="e16da9771721620bbd31c0ecda2c81095f716e4ff819726fd8f57416105118db" Nov 03 20:31:00.446886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.446840 4457 scope.go:122] "RemoveContainer" containerID="e16da9771721620bbd31c0ecda2c81095f716e4ff819726fd8f57416105118db" Nov 03 20:31:00.447901 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:31:00.447845 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16da9771721620bbd31c0ecda2c81095f716e4ff819726fd8f57416105118db\": container with ID starting with e16da9771721620bbd31c0ecda2c81095f716e4ff819726fd8f57416105118db not found: ID does not exist" containerID="e16da9771721620bbd31c0ecda2c81095f716e4ff819726fd8f57416105118db" Nov 03 20:31:00.448071 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.448042 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16da9771721620bbd31c0ecda2c81095f716e4ff819726fd8f57416105118db"} err="failed to get container status \"e16da9771721620bbd31c0ecda2c81095f716e4ff819726fd8f57416105118db\": rpc error: code = NotFound desc = could not find container \"e16da9771721620bbd31c0ecda2c81095f716e4ff819726fd8f57416105118db\": container with ID starting with e16da9771721620bbd31c0ecda2c81095f716e4ff819726fd8f57416105118db not found: ID does not exist" Nov 03 20:31:00.473579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.473548 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn2fp\" (UniqueName: \"kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-kube-api-access-zn2fp\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.473719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.473583 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.473719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.473596 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.473719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.473605 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.473719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.473614 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab0fbd33-7b13-4702-b80d-a102f28187be-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.473719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.473622 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab0fbd33-7b13-4702-b80d-a102f28187be-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.473719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.473630 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab0fbd33-7b13-4702-b80d-a102f28187be-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:00.474979 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.474960 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh"] Nov 03 20:31:00.482920 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:00.482890 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8013d9-openshift-default-cd89f7485-86jqh"] Nov 03 20:31:01.763616 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:01.763578 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" path="/var/lib/kubelet/pods/ab0fbd33-7b13-4702-b80d-a102f28187be/volumes" Nov 03 20:31:06.149564 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:31:06.149524 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:31:07.347633 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:07.347601 4457 scope.go:122] "RemoveContainer" containerID="7d40943aad01781caad8c08fa2423b2cc04bfebf682487b86da0899c410e2a40" Nov 03 20:31:19.428845 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.428812 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh"] Nov 03 20:31:19.429651 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429634 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" containerName="cleanup" Nov 03 20:31:19.429651 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429652 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" containerName="cleanup" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429669 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" containerName="cleanup" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429675 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" containerName="cleanup" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429684 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d168551a-ced0-4313-83f4-6f1d4f3b758d" containerName="cleanup" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429690 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d168551a-ced0-4313-83f4-6f1d4f3b758d" containerName="cleanup" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429698 4457 cpu_manager.go:414] "containerMap: removing container" podUID="b1023951-0cef-4afe-b7d8-76d8a9775abb" containerName="revoke-keys" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429703 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1023951-0cef-4afe-b7d8-76d8a9775abb" containerName="revoke-keys" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429711 4457 cpu_manager.go:414] "containerMap: removing container" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429716 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429724 4457 cpu_manager.go:414] "containerMap: removing container" podUID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" containerName="cleanup" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429729 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" containerName="cleanup" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429736 4457 cpu_manager.go:414] "containerMap: removing container" podUID="3bc0361e-b362-496f-8d29-ac11c012caa6" containerName="payload-pre-processing" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429742 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc0361e-b362-496f-8d29-ac11c012caa6" containerName="payload-pre-processing" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429754 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" containerName="cleanup" Nov 03 20:31:19.429759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429762 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429773 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429777 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429783 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429787 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429797 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d168551a-ced0-4313-83f4-6f1d4f3b758d" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429801 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d168551a-ced0-4313-83f4-6f1d4f3b758d" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429809 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429813 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429819 4457 cpu_manager.go:414] "containerMap: removing container" podUID="7f4de79d-18ac-4f57-803e-fb09ddcc8134" containerName="revoke-keys" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429825 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4de79d-18ac-4f57-803e-fb09ddcc8134" containerName="revoke-keys" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429839 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429844 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429851 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerName="istio-proxy" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429855 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerName="istio-proxy" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429869 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d168551a-ced0-4313-83f4-6f1d4f3b758d" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429875 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d168551a-ced0-4313-83f4-6f1d4f3b758d" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429881 4457 cpu_manager.go:414] "containerMap: removing container" podUID="f52a0620-190b-418a-afdd-8c8ac053a1d4" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429885 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52a0620-190b-418a-afdd-8c8ac053a1d4" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429893 4457 cpu_manager.go:414] "containerMap: removing container" podUID="f52a0620-190b-418a-afdd-8c8ac053a1d4" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429898 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52a0620-190b-418a-afdd-8c8ac053a1d4" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429912 4457 cpu_manager.go:414] "containerMap: removing container" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429917 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429926 4457 cpu_manager.go:414] "containerMap: removing container" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429930 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429943 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429947 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429957 4457 cpu_manager.go:414] "containerMap: removing container" podUID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.429961 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430083 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6387668-070a-49e7-ab69-62ca763db030" containerName="main" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430092 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c925c7-63db-46a9-a936-a0a56f28e396" containerName="istio-proxy" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430098 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430105 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430110 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430117 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430123 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52a0620-190b-418a-afdd-8c8ac053a1d4" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430130 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430137 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430143 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="d168551a-ced0-4313-83f4-6f1d4f3b758d" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430148 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1023951-0cef-4afe-b7d8-76d8a9775abb" containerName="revoke-keys" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430155 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="555064e0-9ca2-47de-b1c0-8473f3729219" containerName="main" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430161 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc0361e-b362-496f-8d29-ac11c012caa6" containerName="payload-pre-processing" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430168 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430173 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c5f7ba-2be4-4ca4-9223-7923c9ad9121" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430178 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430184 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="d168551a-ced0-4313-83f4-6f1d4f3b758d" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430189 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430195 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4de79d-18ac-4f57-803e-fb09ddcc8134" containerName="revoke-keys" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430202 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9fce91d-067e-413a-8d5b-85d5b3d04e80" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430208 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab0fbd33-7b13-4702-b80d-a102f28187be" containerName="istio-proxy" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430213 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52a0620-190b-418a-afdd-8c8ac053a1d4" containerName="cleanup" Nov 03 20:31:19.430197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.430220 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52a0620-190b-418a-afdd-8c8ac053a1d4" containerName="cleanup" Nov 03 20:31:19.437046 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.437021 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" Nov 03 20:31:19.441487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.441466 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:31:19.442621 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.442596 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:31:19.443005 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.442981 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh"] Nov 03 20:31:19.529172 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.529137 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-861df19a-2mqzh\" (UID: \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" Nov 03 20:31:19.529340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.529316 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-861df19a-2mqzh\" (UID: \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" Nov 03 20:31:19.529407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.529348 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpljb\" (UniqueName: \"kubernetes.io/projected/9b812c57-ec4a-44e0-bc19-f5c2498ed507-kube-api-access-fpljb\") pod \"maas-api-revoke-sub-861df19a-2mqzh\" (UID: \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" Nov 03 20:31:19.631378 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.631344 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-861df19a-2mqzh\" (UID: \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" Nov 03 20:31:19.631569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.631517 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-861df19a-2mqzh\" (UID: \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" Nov 03 20:31:19.631569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.631540 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpljb\" (UniqueName: \"kubernetes.io/projected/9b812c57-ec4a-44e0-bc19-f5c2498ed507-kube-api-access-fpljb\") pod \"maas-api-revoke-sub-861df19a-2mqzh\" (UID: \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" Nov 03 20:31:19.632179 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.632143 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-861df19a-2mqzh\" (UID: \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" Nov 03 20:31:19.634056 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.634031 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-861df19a-2mqzh\" (UID: \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" Nov 03 20:31:19.644302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.644275 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpljb\" (UniqueName: \"kubernetes.io/projected/9b812c57-ec4a-44e0-bc19-f5c2498ed507-kube-api-access-fpljb\") pod \"maas-api-revoke-sub-861df19a-2mqzh\" (UID: \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" Nov 03 20:31:19.755399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:19.755305 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" Nov 03 20:31:20.059433 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:31:20.056479 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b812c57_ec4a_44e0_bc19_f5c2498ed507.slice/crio-b237ad5d4819dfc0bd8527fd172f238cd69db6328de5417c65afd00330ed957b WatchSource:0}: Error finding container b237ad5d4819dfc0bd8527fd172f238cd69db6328de5417c65afd00330ed957b: Status 404 returned error can't find the container with id b237ad5d4819dfc0bd8527fd172f238cd69db6328de5417c65afd00330ed957b Nov 03 20:31:20.059433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:20.058057 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh"] Nov 03 20:31:20.686403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:20.685205 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" event={"ID":"9b812c57-ec4a-44e0-bc19-f5c2498ed507","Type":"ContainerStarted","Data":"9ed3e9089c0f6a1019b07d8ebffbb467d75199a2094370ed4ecc0205c1200b70"} Nov 03 20:31:20.686403 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:20.685327 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" event={"ID":"9b812c57-ec4a-44e0-bc19-f5c2498ed507","Type":"ContainerStarted","Data":"b237ad5d4819dfc0bd8527fd172f238cd69db6328de5417c65afd00330ed957b"} Nov 03 20:31:20.705691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:20.705637 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" podStartSLOduration=1.705623003 podStartE2EDuration="1.705623003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:20.704413523 +0000 UTC m=+2059.080010267" watchObservedRunningTime="2026-11-03 20:31:20.705623003 +0000 UTC m=+2059.081219751" Nov 03 20:31:21.716357 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:21.716318 4457 generic.go:356] "Generic (PLEG): container finished" podID="9b812c57-ec4a-44e0-bc19-f5c2498ed507" containerID="9ed3e9089c0f6a1019b07d8ebffbb467d75199a2094370ed4ecc0205c1200b70" exitCode=0 Nov 03 20:31:21.716847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:21.716439 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" event={"ID":"9b812c57-ec4a-44e0-bc19-f5c2498ed507","Type":"ContainerDied","Data":"9ed3e9089c0f6a1019b07d8ebffbb467d75199a2094370ed4ecc0205c1200b70"} Nov 03 20:31:22.916652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:22.916615 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" Nov 03 20:31:23.007617 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:23.000885 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-openshift-ca\") pod \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\" (UID: \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\") " Nov 03 20:31:23.007617 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:23.000986 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-xks-ca\") pod \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\" (UID: \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\") " Nov 03 20:31:23.007617 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:23.001341 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b812c57-ec4a-44e0-bc19-f5c2498ed507-kube-api-access-fpljb\" (UniqueName: \"kubernetes.io/projected/9b812c57-ec4a-44e0-bc19-f5c2498ed507-kube-api-access-fpljb\") pod \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\" (UID: \"9b812c57-ec4a-44e0-bc19-f5c2498ed507\") " Nov 03 20:31:23.007617 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:23.003666 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-openshift-ca" pod "9b812c57-ec4a-44e0-bc19-f5c2498ed507" (UID: "9b812c57-ec4a-44e0-bc19-f5c2498ed507"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:23.014939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:23.014824 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-xks-ca" pod "9b812c57-ec4a-44e0-bc19-f5c2498ed507" (UID: "9b812c57-ec4a-44e0-bc19-f5c2498ed507"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:23.014939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:23.014908 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b812c57-ec4a-44e0-bc19-f5c2498ed507-kube-api-access-fpljb" pod "9b812c57-ec4a-44e0-bc19-f5c2498ed507" (UID: "9b812c57-ec4a-44e0-bc19-f5c2498ed507"). InnerVolumeSpecName "kube-api-access-fpljb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:23.104443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:23.104404 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:23.104443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:23.104439 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b812c57-ec4a-44e0-bc19-f5c2498ed507-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:23.104661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:23.104454 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpljb\" (UniqueName: \"kubernetes.io/projected/9b812c57-ec4a-44e0-bc19-f5c2498ed507-kube-api-access-fpljb\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:23.737757 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:23.737722 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" event={"ID":"9b812c57-ec4a-44e0-bc19-f5c2498ed507","Type":"ContainerDied","Data":"b237ad5d4819dfc0bd8527fd172f238cd69db6328de5417c65afd00330ed957b"} Nov 03 20:31:23.737986 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:23.737775 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b237ad5d4819dfc0bd8527fd172f238cd69db6328de5417c65afd00330ed957b" Nov 03 20:31:23.737986 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:23.737833 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh" Nov 03 20:31:30.570465 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.570420 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc"] Nov 03 20:31:30.571736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.571715 4457 cpu_manager.go:414] "containerMap: removing container" podUID="9b812c57-ec4a-44e0-bc19-f5c2498ed507" containerName="revoke-keys" Nov 03 20:31:30.571811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.571738 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b812c57-ec4a-44e0-bc19-f5c2498ed507" containerName="revoke-keys" Nov 03 20:31:30.571811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.571763 4457 cpu_manager.go:414] "containerMap: removing container" podUID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" containerName="cleanup" Nov 03 20:31:30.571811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.571771 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" containerName="cleanup" Nov 03 20:31:30.571811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.571790 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" containerName="cleanup" Nov 03 20:31:30.571811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.571798 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8d2d9a-946c-4e76-b319-c7d653ac2306" containerName="cleanup" Nov 03 20:31:30.571969 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.571828 4457 cpu_manager.go:414] "containerMap: removing container" podUID="f52a0620-190b-418a-afdd-8c8ac053a1d4" containerName="cleanup" Nov 03 20:31:30.571969 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.571835 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52a0620-190b-418a-afdd-8c8ac053a1d4" containerName="cleanup" Nov 03 20:31:30.572053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.572041 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea2c970-9b94-45c7-b8e7-e1263a11f8f1" containerName="cleanup" Nov 03 20:31:30.572091 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.572064 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b812c57-ec4a-44e0-bc19-f5c2498ed507" containerName="revoke-keys" Nov 03 20:31:30.572091 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.572077 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="d168551a-ced0-4313-83f4-6f1d4f3b758d" containerName="cleanup" Nov 03 20:31:30.584444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.584367 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" Nov 03 20:31:30.589015 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.588409 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:31:30.589627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.589592 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:31:30.590176 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.590153 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc"] Nov 03 20:31:30.677829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.677793 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb79g\" (UniqueName: \"kubernetes.io/projected/3bd78a7f-4170-42c1-9bb6-919b7327325d-kube-api-access-tb79g\") pod \"maas-api-revoke-sub-f757389a-24stc\" (UID: \"3bd78a7f-4170-42c1-9bb6-919b7327325d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" Nov 03 20:31:30.678142 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.677852 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f757389a-24stc\" (UID: \"3bd78a7f-4170-42c1-9bb6-919b7327325d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" Nov 03 20:31:30.678142 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.677993 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f757389a-24stc\" (UID: \"3bd78a7f-4170-42c1-9bb6-919b7327325d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" Nov 03 20:31:30.779610 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.779570 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb79g\" (UniqueName: \"kubernetes.io/projected/3bd78a7f-4170-42c1-9bb6-919b7327325d-kube-api-access-tb79g\") pod \"maas-api-revoke-sub-f757389a-24stc\" (UID: \"3bd78a7f-4170-42c1-9bb6-919b7327325d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" Nov 03 20:31:30.779810 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.779637 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f757389a-24stc\" (UID: \"3bd78a7f-4170-42c1-9bb6-919b7327325d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" Nov 03 20:31:30.779810 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.779709 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f757389a-24stc\" (UID: \"3bd78a7f-4170-42c1-9bb6-919b7327325d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" Nov 03 20:31:30.780291 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.780269 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f757389a-24stc\" (UID: \"3bd78a7f-4170-42c1-9bb6-919b7327325d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" Nov 03 20:31:30.782138 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.782114 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f757389a-24stc\" (UID: \"3bd78a7f-4170-42c1-9bb6-919b7327325d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" Nov 03 20:31:30.791760 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.791738 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb79g\" (UniqueName: \"kubernetes.io/projected/3bd78a7f-4170-42c1-9bb6-919b7327325d-kube-api-access-tb79g\") pod \"maas-api-revoke-sub-f757389a-24stc\" (UID: \"3bd78a7f-4170-42c1-9bb6-919b7327325d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" Nov 03 20:31:30.917020 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:30.916938 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" Nov 03 20:31:31.219437 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:31:31.219363 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd78a7f_4170_42c1_9bb6_919b7327325d.slice/crio-d9c2f21295d8e5ecaa0de1a000b7b7fb06d81b31a2ca13a22c52aa1546ef5d1d WatchSource:0}: Error finding container d9c2f21295d8e5ecaa0de1a000b7b7fb06d81b31a2ca13a22c52aa1546ef5d1d: Status 404 returned error can't find the container with id d9c2f21295d8e5ecaa0de1a000b7b7fb06d81b31a2ca13a22c52aa1546ef5d1d Nov 03 20:31:31.220148 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:31.220123 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc"] Nov 03 20:31:31.825428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:31.825347 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" event={"ID":"3bd78a7f-4170-42c1-9bb6-919b7327325d","Type":"ContainerStarted","Data":"8c565f054a633828b6d0fd16eea5f29ac27c70741d18ce72bbe7b947c0a1f648"} Nov 03 20:31:31.825832 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:31.825464 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" event={"ID":"3bd78a7f-4170-42c1-9bb6-919b7327325d","Type":"ContainerStarted","Data":"d9c2f21295d8e5ecaa0de1a000b7b7fb06d81b31a2ca13a22c52aa1546ef5d1d"} Nov 03 20:31:31.845283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:31.845232 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" podStartSLOduration=1.845220173 podStartE2EDuration="1.845220173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:31:31.844551294 +0000 UTC m=+2070.220148040" watchObservedRunningTime="2026-11-03 20:31:31.845220173 +0000 UTC m=+2070.220816919" Nov 03 20:31:32.836032 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:32.835993 4457 generic.go:356] "Generic (PLEG): container finished" podID="3bd78a7f-4170-42c1-9bb6-919b7327325d" containerID="8c565f054a633828b6d0fd16eea5f29ac27c70741d18ce72bbe7b947c0a1f648" exitCode=0 Nov 03 20:31:32.836502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:32.836035 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" event={"ID":"3bd78a7f-4170-42c1-9bb6-919b7327325d","Type":"ContainerDied","Data":"8c565f054a633828b6d0fd16eea5f29ac27c70741d18ce72bbe7b947c0a1f648"} Nov 03 20:31:34.049762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:34.049739 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" Nov 03 20:31:34.143739 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:34.143642 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-openshift-ca\") pod \"3bd78a7f-4170-42c1-9bb6-919b7327325d\" (UID: \"3bd78a7f-4170-42c1-9bb6-919b7327325d\") " Nov 03 20:31:34.143918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:34.143847 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd78a7f-4170-42c1-9bb6-919b7327325d-kube-api-access-tb79g\" (UniqueName: \"kubernetes.io/projected/3bd78a7f-4170-42c1-9bb6-919b7327325d-kube-api-access-tb79g\") pod \"3bd78a7f-4170-42c1-9bb6-919b7327325d\" (UID: \"3bd78a7f-4170-42c1-9bb6-919b7327325d\") " Nov 03 20:31:34.143994 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:34.143927 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-xks-ca\") pod \"3bd78a7f-4170-42c1-9bb6-919b7327325d\" (UID: \"3bd78a7f-4170-42c1-9bb6-919b7327325d\") " Nov 03 20:31:34.144072 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:34.144043 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-openshift-ca" pod "3bd78a7f-4170-42c1-9bb6-919b7327325d" (UID: "3bd78a7f-4170-42c1-9bb6-919b7327325d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:34.144854 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:34.144825 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:34.146272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:34.146236 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd78a7f-4170-42c1-9bb6-919b7327325d-kube-api-access-tb79g" pod "3bd78a7f-4170-42c1-9bb6-919b7327325d" (UID: "3bd78a7f-4170-42c1-9bb6-919b7327325d"). InnerVolumeSpecName "kube-api-access-tb79g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:34.146748 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:34.146716 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-xks-ca" pod "3bd78a7f-4170-42c1-9bb6-919b7327325d" (UID: "3bd78a7f-4170-42c1-9bb6-919b7327325d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:34.264928 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:34.263896 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb79g\" (UniqueName: \"kubernetes.io/projected/3bd78a7f-4170-42c1-9bb6-919b7327325d-kube-api-access-tb79g\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:34.264928 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:34.263934 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd78a7f-4170-42c1-9bb6-919b7327325d-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:34.858706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:34.858673 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" Nov 03 20:31:34.858706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:34.858693 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc" event={"ID":"3bd78a7f-4170-42c1-9bb6-919b7327325d","Type":"ContainerDied","Data":"d9c2f21295d8e5ecaa0de1a000b7b7fb06d81b31a2ca13a22c52aa1546ef5d1d"} Nov 03 20:31:34.858950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:34.858754 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9c2f21295d8e5ecaa0de1a000b7b7fb06d81b31a2ca13a22c52aa1546ef5d1d" Nov 03 20:31:34.915440 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:31:34.915403 4457 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd78a7f_4170_42c1_9bb6_919b7327325d.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:31:36.149499 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:31:36.149460 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:31:41.734243 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:41.734190 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx"] Nov 03 20:31:42.085287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.085250 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj"] Nov 03 20:31:42.086846 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.086816 4457 cpu_manager.go:414] "containerMap: removing container" podUID="3bd78a7f-4170-42c1-9bb6-919b7327325d" containerName="revoke-keys" Nov 03 20:31:42.087011 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.086998 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd78a7f-4170-42c1-9bb6-919b7327325d" containerName="revoke-keys" Nov 03 20:31:42.087433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.087413 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd78a7f-4170-42c1-9bb6-919b7327325d" containerName="revoke-keys" Nov 03 20:31:42.096938 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.096916 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" Nov 03 20:31:42.098650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.098603 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj"] Nov 03 20:31:42.101758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.101730 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:31:42.101944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.101893 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:31:42.209358 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.209321 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj\" (UID: \"923c18ff-2848-495b-836a-ae922f1880d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" Nov 03 20:31:42.209584 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.209375 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj\" (UID: \"923c18ff-2848-495b-836a-ae922f1880d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" Nov 03 20:31:42.209766 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.209741 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm84z\" (UniqueName: \"kubernetes.io/projected/923c18ff-2848-495b-836a-ae922f1880d4-kube-api-access-bm84z\") pod \"maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj\" (UID: \"923c18ff-2848-495b-836a-ae922f1880d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" Nov 03 20:31:42.313409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.311593 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj\" (UID: \"923c18ff-2848-495b-836a-ae922f1880d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" Nov 03 20:31:42.313409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.311939 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm84z\" (UniqueName: \"kubernetes.io/projected/923c18ff-2848-495b-836a-ae922f1880d4-kube-api-access-bm84z\") pod \"maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj\" (UID: \"923c18ff-2848-495b-836a-ae922f1880d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" Nov 03 20:31:42.313409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.312164 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj\" (UID: \"923c18ff-2848-495b-836a-ae922f1880d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" Nov 03 20:31:42.313409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.312949 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj\" (UID: \"923c18ff-2848-495b-836a-ae922f1880d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" Nov 03 20:31:42.321963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.321926 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj\" (UID: \"923c18ff-2848-495b-836a-ae922f1880d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" Nov 03 20:31:42.327516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.327453 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm84z\" (UniqueName: \"kubernetes.io/projected/923c18ff-2848-495b-836a-ae922f1880d4-kube-api-access-bm84z\") pod \"maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj\" (UID: \"923c18ff-2848-495b-836a-ae922f1880d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" Nov 03 20:31:42.422955 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.422877 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" Nov 03 20:31:42.718756 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:31:42.718713 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923c18ff_2848_495b_836a_ae922f1880d4.slice/crio-7957d6ca6fa3f53a2da61f708a80b241910b4b33440e3b9c8ef435e54d0f5fe3 WatchSource:0}: Error finding container 7957d6ca6fa3f53a2da61f708a80b241910b4b33440e3b9c8ef435e54d0f5fe3: Status 404 returned error can't find the container with id 7957d6ca6fa3f53a2da61f708a80b241910b4b33440e3b9c8ef435e54d0f5fe3 Nov 03 20:31:42.720766 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.720630 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj"] Nov 03 20:31:42.992203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:42.992154 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" event={"ID":"923c18ff-2848-495b-836a-ae922f1880d4","Type":"ContainerStarted","Data":"7957d6ca6fa3f53a2da61f708a80b241910b4b33440e3b9c8ef435e54d0f5fe3"} Nov 03 20:31:44.016072 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:44.015984 4457 generic.go:356] "Generic (PLEG): container finished" podID="923c18ff-2848-495b-836a-ae922f1880d4" containerID="4244b9ffdd1e958d07ebaeec1b5e4ac72be4c12aba54d0f76fcfe2a54b209128" exitCode=0 Nov 03 20:31:44.016496 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:44.016074 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" event={"ID":"923c18ff-2848-495b-836a-ae922f1880d4","Type":"ContainerDied","Data":"4244b9ffdd1e958d07ebaeec1b5e4ac72be4c12aba54d0f76fcfe2a54b209128"} Nov 03 20:31:45.232489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:45.232421 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" Nov 03 20:31:45.377511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:45.377478 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-openshift-ca\") pod \"923c18ff-2848-495b-836a-ae922f1880d4\" (UID: \"923c18ff-2848-495b-836a-ae922f1880d4\") " Nov 03 20:31:45.377704 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:45.377635 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/923c18ff-2848-495b-836a-ae922f1880d4-kube-api-access-bm84z\" (UniqueName: \"kubernetes.io/projected/923c18ff-2848-495b-836a-ae922f1880d4-kube-api-access-bm84z\") pod \"923c18ff-2848-495b-836a-ae922f1880d4\" (UID: \"923c18ff-2848-495b-836a-ae922f1880d4\") " Nov 03 20:31:45.377704 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:45.377665 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-xks-ca\") pod \"923c18ff-2848-495b-836a-ae922f1880d4\" (UID: \"923c18ff-2848-495b-836a-ae922f1880d4\") " Nov 03 20:31:45.377901 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:45.377873 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-openshift-ca" pod "923c18ff-2848-495b-836a-ae922f1880d4" (UID: "923c18ff-2848-495b-836a-ae922f1880d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:31:45.378353 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:45.378331 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:45.379975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:45.379951 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-xks-ca" pod "923c18ff-2848-495b-836a-ae922f1880d4" (UID: "923c18ff-2848-495b-836a-ae922f1880d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:45.380069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:45.379955 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/923c18ff-2848-495b-836a-ae922f1880d4-kube-api-access-bm84z" pod "923c18ff-2848-495b-836a-ae922f1880d4" (UID: "923c18ff-2848-495b-836a-ae922f1880d4"). InnerVolumeSpecName "kube-api-access-bm84z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:45.480989 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:45.480726 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm84z\" (UniqueName: \"kubernetes.io/projected/923c18ff-2848-495b-836a-ae922f1880d4-kube-api-access-bm84z\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:45.480989 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:45.480771 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/923c18ff-2848-495b-836a-ae922f1880d4-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:46.035497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:46.035409 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" Nov 03 20:31:46.035497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:46.035452 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj" event={"ID":"923c18ff-2848-495b-836a-ae922f1880d4","Type":"ContainerDied","Data":"7957d6ca6fa3f53a2da61f708a80b241910b4b33440e3b9c8ef435e54d0f5fe3"} Nov 03 20:31:46.035685 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:46.035502 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7957d6ca6fa3f53a2da61f708a80b241910b4b33440e3b9c8ef435e54d0f5fe3" Nov 03 20:31:52.214672 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:52.214632 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl"] Nov 03 20:31:52.220824 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:52.220791 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx"] Nov 03 20:31:56.770599 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:56.770552 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" podUID="5d2da67e-e41e-441b-9a9d-1272bc5726a4" containerName="main" containerID="cri-o://7437236e4643c954e24c4946b461905d569c6cee339db97f79a9653e89207ec1" gracePeriod=45 Nov 03 20:31:57.096596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.096568 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:31:57.137515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.137483 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-dshm\") pod \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " Nov 03 20:31:57.137707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.137541 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tls-certs\") pod \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " Nov 03 20:31:57.137707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.137628 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tmp-dir\") pod \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " Nov 03 20:31:57.137842 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.137733 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-home\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-home\") pod \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " Nov 03 20:31:57.137842 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.137767 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-model-cache\") pod \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " Nov 03 20:31:57.137992 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.137841 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d2da67e-e41e-441b-9a9d-1272bc5726a4-kube-api-access-m8988\" (UniqueName: \"kubernetes.io/projected/5d2da67e-e41e-441b-9a9d-1272bc5726a4-kube-api-access-m8988\") pod \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\" (UID: \"5d2da67e-e41e-441b-9a9d-1272bc5726a4\") " Nov 03 20:31:57.138071 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.138036 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-home" pod "5d2da67e-e41e-441b-9a9d-1272bc5726a4" (UID: "5d2da67e-e41e-441b-9a9d-1272bc5726a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:57.138132 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.138049 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-model-cache" pod "5d2da67e-e41e-441b-9a9d-1272bc5726a4" (UID: "5d2da67e-e41e-441b-9a9d-1272bc5726a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:57.138430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.138405 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tmp-dir" pod "5d2da67e-e41e-441b-9a9d-1272bc5726a4" (UID: "5d2da67e-e41e-441b-9a9d-1272bc5726a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:57.138700 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.138679 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:57.138700 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.138700 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:57.138890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.138714 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:57.140558 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.140451 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tls-certs" pod "5d2da67e-e41e-441b-9a9d-1272bc5726a4" (UID: "5d2da67e-e41e-441b-9a9d-1272bc5726a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:31:57.141585 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.141559 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-dshm" pod "5d2da67e-e41e-441b-9a9d-1272bc5726a4" (UID: "5d2da67e-e41e-441b-9a9d-1272bc5726a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:31:57.141797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.141775 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d2da67e-e41e-441b-9a9d-1272bc5726a4-kube-api-access-m8988" pod "5d2da67e-e41e-441b-9a9d-1272bc5726a4" (UID: "5d2da67e-e41e-441b-9a9d-1272bc5726a4"). InnerVolumeSpecName "kube-api-access-m8988". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:31:57.162519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.162481 4457 generic.go:356] "Generic (PLEG): container finished" podID="5d2da67e-e41e-441b-9a9d-1272bc5726a4" containerID="7437236e4643c954e24c4946b461905d569c6cee339db97f79a9653e89207ec1" exitCode=0 Nov 03 20:31:57.162691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.162672 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" event={"ID":"5d2da67e-e41e-441b-9a9d-1272bc5726a4","Type":"ContainerDied","Data":"7437236e4643c954e24c4946b461905d569c6cee339db97f79a9653e89207ec1"} Nov 03 20:31:57.162753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.162711 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" event={"ID":"5d2da67e-e41e-441b-9a9d-1272bc5726a4","Type":"ContainerDied","Data":"c7a60dbae27829718d43a3703fb54228a544c1c7b559e2662b669f3a58990e9f"} Nov 03 20:31:57.162753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.162734 4457 scope.go:122] "RemoveContainer" containerID="7437236e4643c954e24c4946b461905d569c6cee339db97f79a9653e89207ec1" Nov 03 20:31:57.162876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.162864 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx" Nov 03 20:31:57.203898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.203577 4457 scope.go:122] "RemoveContainer" containerID="7437236e4643c954e24c4946b461905d569c6cee339db97f79a9653e89207ec1" Nov 03 20:31:57.204232 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:31:57.204139 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7437236e4643c954e24c4946b461905d569c6cee339db97f79a9653e89207ec1\": container with ID starting with 7437236e4643c954e24c4946b461905d569c6cee339db97f79a9653e89207ec1 not found: ID does not exist" containerID="7437236e4643c954e24c4946b461905d569c6cee339db97f79a9653e89207ec1" Nov 03 20:31:57.204232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.204176 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7437236e4643c954e24c4946b461905d569c6cee339db97f79a9653e89207ec1"} err="failed to get container status \"7437236e4643c954e24c4946b461905d569c6cee339db97f79a9653e89207ec1\": rpc error: code = NotFound desc = could not find container \"7437236e4643c954e24c4946b461905d569c6cee339db97f79a9653e89207ec1\": container with ID starting with 7437236e4643c954e24c4946b461905d569c6cee339db97f79a9653e89207ec1 not found: ID does not exist" Nov 03 20:31:57.209534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.209506 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx"] Nov 03 20:31:57.214902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.214877 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-34e844/test-model-34e844-kserve-6945d4458c-4rjtx"] Nov 03 20:31:57.214992 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.214934 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" podUID="130c5e5b-43ee-4329-ab13-f37b6c3981fb" containerName="payload-processing" containerID="cri-o://b3c4f49d657b41cbb403533d2a66b6d862a14b489768801f0c1b628b0fecc44f" gracePeriod=25 Nov 03 20:31:57.221534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.221502 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" podUID="9e5dbdf1-f78c-4368-aabb-232578d293a7" containerName="payload-pre-processing" containerID="cri-o://de36493ec97096c2941d816c9d0c5b2905711a7f834f593a9a27a7bde209baaf" gracePeriod=25 Nov 03 20:31:57.240631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.240593 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8988\" (UniqueName: \"kubernetes.io/projected/5d2da67e-e41e-441b-9a9d-1272bc5726a4-kube-api-access-m8988\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:57.240631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.240625 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d2da67e-e41e-441b-9a9d-1272bc5726a4-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:57.240631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.240637 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d2da67e-e41e-441b-9a9d-1272bc5726a4-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:31:57.763282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:31:57.763243 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d2da67e-e41e-441b-9a9d-1272bc5726a4" path="/var/lib/kubelet/pods/5d2da67e-e41e-441b-9a9d-1272bc5726a4/volumes" Nov 03 20:32:02.805299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:02.805263 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:32:02.809758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:02.809731 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:32:02.833932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:02.833897 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:32:02.838939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:02.838905 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:32:03.025586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:03.025548 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" podUID="9e5dbdf1-f78c-4368-aabb-232578d293a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:32:03.025761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:03.025598 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" podUID="130c5e5b-43ee-4329-ab13-f37b6c3981fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:32:03.540472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:03.540436 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf"] Nov 03 20:32:03.540721 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:03.540684 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" containerID="cri-o://6478cfa8470743528dc74fde5bcc0d8ec021a4a6e5f596473acbabb2d1aaecdb" gracePeriod=30 Nov 03 20:32:06.149528 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:32:06.149486 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:32:07.494707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:07.494670 4457 scope.go:122] "RemoveContainer" containerID="978e42784495c709eec455d0195d98fdebfa842d900fef834c07d84a557d74f9" Nov 03 20:32:07.506942 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:07.506918 4457 scope.go:122] "RemoveContainer" containerID="21367622f69c2b9da0e2954ed77bb4886acaf7f66c98cbbb6317b5b219e61943" Nov 03 20:32:08.881777 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.881749 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:32:08.889034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.889003 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-podinfo\") pod \"b5721044-7f44-4a81-a212-abc21f39bc1e\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " Nov 03 20:32:08.889191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.889090 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-credential-socket\") pod \"b5721044-7f44-4a81-a212-abc21f39bc1e\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " Nov 03 20:32:08.889259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.889217 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-socket\") pod \"b5721044-7f44-4a81-a212-abc21f39bc1e\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " Nov 03 20:32:08.889315 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.889284 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-certs\") pod \"b5721044-7f44-4a81-a212-abc21f39bc1e\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " Nov 03 20:32:08.889380 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.889347 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-envoy\") pod \"b5721044-7f44-4a81-a212-abc21f39bc1e\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " Nov 03 20:32:08.889497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.889377 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-kube-api-access-r96h9\" (UniqueName: \"kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-kube-api-access-r96h9\") pod \"b5721044-7f44-4a81-a212-abc21f39bc1e\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " Nov 03 20:32:08.889497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.889433 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-credential-socket" pod "b5721044-7f44-4a81-a212-abc21f39bc1e" (UID: "b5721044-7f44-4a81-a212-abc21f39bc1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:08.889497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.889476 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-token\" (UniqueName: \"kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-token\") pod \"b5721044-7f44-4a81-a212-abc21f39bc1e\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " Nov 03 20:32:08.889661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.889564 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-data\") pod \"b5721044-7f44-4a81-a212-abc21f39bc1e\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " Nov 03 20:32:08.889661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.889653 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-socket" pod "b5721044-7f44-4a81-a212-abc21f39bc1e" (UID: "b5721044-7f44-4a81-a212-abc21f39bc1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:08.889778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.889690 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5721044-7f44-4a81-a212-abc21f39bc1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5721044-7f44-4a81-a212-abc21f39bc1e-istiod-ca-cert\") pod \"b5721044-7f44-4a81-a212-abc21f39bc1e\" (UID: \"b5721044-7f44-4a81-a212-abc21f39bc1e\") " Nov 03 20:32:08.889844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.889824 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-data" pod "b5721044-7f44-4a81-a212-abc21f39bc1e" (UID: "b5721044-7f44-4a81-a212-abc21f39bc1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:08.890190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.890164 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-certs" pod "b5721044-7f44-4a81-a212-abc21f39bc1e" (UID: "b5721044-7f44-4a81-a212-abc21f39bc1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:08.890295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.890274 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5721044-7f44-4a81-a212-abc21f39bc1e-istiod-ca-cert" pod "b5721044-7f44-4a81-a212-abc21f39bc1e" (UID: "b5721044-7f44-4a81-a212-abc21f39bc1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:08.890504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.890461 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:08.890596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.890510 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:08.890596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.890550 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:08.890596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.890570 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:08.890596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.890583 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5721044-7f44-4a81-a212-abc21f39bc1e-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:08.892573 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.892153 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-podinfo" pod "b5721044-7f44-4a81-a212-abc21f39bc1e" (UID: "b5721044-7f44-4a81-a212-abc21f39bc1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:08.892573 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.892169 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-kube-api-access-r96h9" pod "b5721044-7f44-4a81-a212-abc21f39bc1e" (UID: "b5721044-7f44-4a81-a212-abc21f39bc1e"). InnerVolumeSpecName "kube-api-access-r96h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:08.892751 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.892719 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-envoy" pod "b5721044-7f44-4a81-a212-abc21f39bc1e" (UID: "b5721044-7f44-4a81-a212-abc21f39bc1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:08.893517 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.893491 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-token" pod "b5721044-7f44-4a81-a212-abc21f39bc1e" (UID: "b5721044-7f44-4a81-a212-abc21f39bc1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:08.992341 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.992307 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:08.992341 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.992342 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r96h9\" (UniqueName: \"kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-kube-api-access-r96h9\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:08.992568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.992358 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:08.992568 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:08.992372 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5721044-7f44-4a81-a212-abc21f39bc1e-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:09.371411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.371343 4457 generic.go:356] "Generic (PLEG): container finished" podID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerID="6478cfa8470743528dc74fde5bcc0d8ec021a4a6e5f596473acbabb2d1aaecdb" exitCode=0 Nov 03 20:32:09.371781 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.371731 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" event={"ID":"b5721044-7f44-4a81-a212-abc21f39bc1e","Type":"ContainerDied","Data":"6478cfa8470743528dc74fde5bcc0d8ec021a4a6e5f596473acbabb2d1aaecdb"} Nov 03 20:32:09.371945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.371799 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" event={"ID":"b5721044-7f44-4a81-a212-abc21f39bc1e","Type":"ContainerDied","Data":"f1d64b4ba0a0b25663b0a1d9969feb3032b66a762c84af828e85220d770750d2"} Nov 03 20:32:09.371945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.371849 4457 scope.go:122] "RemoveContainer" containerID="6478cfa8470743528dc74fde5bcc0d8ec021a4a6e5f596473acbabb2d1aaecdb" Nov 03 20:32:09.372072 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.372039 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf" Nov 03 20:32:09.423241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.423208 4457 scope.go:122] "RemoveContainer" containerID="6478cfa8470743528dc74fde5bcc0d8ec021a4a6e5f596473acbabb2d1aaecdb" Nov 03 20:32:09.423723 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:32:09.423691 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6478cfa8470743528dc74fde5bcc0d8ec021a4a6e5f596473acbabb2d1aaecdb\": container with ID starting with 6478cfa8470743528dc74fde5bcc0d8ec021a4a6e5f596473acbabb2d1aaecdb not found: ID does not exist" containerID="6478cfa8470743528dc74fde5bcc0d8ec021a4a6e5f596473acbabb2d1aaecdb" Nov 03 20:32:09.423842 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.423731 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6478cfa8470743528dc74fde5bcc0d8ec021a4a6e5f596473acbabb2d1aaecdb"} err="failed to get container status \"6478cfa8470743528dc74fde5bcc0d8ec021a4a6e5f596473acbabb2d1aaecdb\": rpc error: code = NotFound desc = could not find container \"6478cfa8470743528dc74fde5bcc0d8ec021a4a6e5f596473acbabb2d1aaecdb\": container with ID starting with 6478cfa8470743528dc74fde5bcc0d8ec021a4a6e5f596473acbabb2d1aaecdb not found: ID does not exist" Nov 03 20:32:09.437041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.437010 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf"] Nov 03 20:32:09.445137 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.445111 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-wtjrf"] Nov 03 20:32:09.629116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.629039 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2"] Nov 03 20:32:09.629971 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.629949 4457 cpu_manager.go:414] "containerMap: removing container" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" Nov 03 20:32:09.630084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.629973 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" Nov 03 20:32:09.630084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.630025 4457 cpu_manager.go:414] "containerMap: removing container" podUID="5d2da67e-e41e-441b-9a9d-1272bc5726a4" containerName="main" Nov 03 20:32:09.630084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.630035 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d2da67e-e41e-441b-9a9d-1272bc5726a4" containerName="main" Nov 03 20:32:09.630196 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.630098 4457 cpu_manager.go:414] "containerMap: removing container" podUID="923c18ff-2848-495b-836a-ae922f1880d4" containerName="revoke-keys" Nov 03 20:32:09.630196 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.630106 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="923c18ff-2848-495b-836a-ae922f1880d4" containerName="revoke-keys" Nov 03 20:32:09.630349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.630334 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d2da67e-e41e-441b-9a9d-1272bc5726a4" containerName="main" Nov 03 20:32:09.630402 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.630362 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="923c18ff-2848-495b-836a-ae922f1880d4" containerName="revoke-keys" Nov 03 20:32:09.630402 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.630376 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" containerName="istio-proxy" Nov 03 20:32:09.635932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.635907 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" Nov 03 20:32:09.641788 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.641763 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2"] Nov 03 20:32:09.704854 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.704821 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c76md\" (UniqueName: \"kubernetes.io/projected/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-kube-api-access-c76md\") pod \"maas-api-revoke-sub-c1220971-4h5d2\" (UID: \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" Nov 03 20:32:09.705040 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.704953 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1220971-4h5d2\" (UID: \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" Nov 03 20:32:09.705099 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.705045 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1220971-4h5d2\" (UID: \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" Nov 03 20:32:09.727510 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.727475 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:32:09.727687 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.727571 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:32:09.763049 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.763003 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5721044-7f44-4a81-a212-abc21f39bc1e" path="/var/lib/kubelet/pods/b5721044-7f44-4a81-a212-abc21f39bc1e/volumes" Nov 03 20:32:09.806584 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.806554 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1220971-4h5d2\" (UID: \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" Nov 03 20:32:09.806813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.806794 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c76md\" (UniqueName: \"kubernetes.io/projected/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-kube-api-access-c76md\") pod \"maas-api-revoke-sub-c1220971-4h5d2\" (UID: \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" Nov 03 20:32:09.806884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.806870 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1220971-4h5d2\" (UID: \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" Nov 03 20:32:09.807309 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.807283 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1220971-4h5d2\" (UID: \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" Nov 03 20:32:09.809495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.809467 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1220971-4h5d2\" (UID: \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" Nov 03 20:32:09.819522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.819496 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c76md\" (UniqueName: \"kubernetes.io/projected/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-kube-api-access-c76md\") pod \"maas-api-revoke-sub-c1220971-4h5d2\" (UID: \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" Nov 03 20:32:09.868717 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.868677 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5"] Nov 03 20:32:09.915221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.915123 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" podUID="e8d055e5-7791-4f91-a52d-314d13289239" containerName="maas-api" containerID="cri-o://88883fc000b94f46e32228a4017f26c61fa9138cae7fcda5b4b5f88ec8593a6f" gracePeriod=30 Nov 03 20:32:09.972431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:09.970363 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" Nov 03 20:32:10.527421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.526800 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj"] Nov 03 20:32:10.546295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.546241 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-77e83b-p57nj"] Nov 03 20:32:10.552646 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.552603 4457 generic.go:356] "Generic (PLEG): container finished" podID="e8d055e5-7791-4f91-a52d-314d13289239" containerID="88883fc000b94f46e32228a4017f26c61fa9138cae7fcda5b4b5f88ec8593a6f" exitCode=0 Nov 03 20:32:10.552902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.552676 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" event={"ID":"e8d055e5-7791-4f91-a52d-314d13289239","Type":"ContainerDied","Data":"88883fc000b94f46e32228a4017f26c61fa9138cae7fcda5b4b5f88ec8593a6f"} Nov 03 20:32:10.722131 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.722088 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2"] Nov 03 20:32:10.731440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.728300 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:32:10.813809 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.813773 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8d055e5-7791-4f91-a52d-314d13289239-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d055e5-7791-4f91-a52d-314d13289239-openshift-service-ca\") pod \"e8d055e5-7791-4f91-a52d-314d13289239\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " Nov 03 20:32:10.814030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.814005 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-maas-api-tls\") pod \"e8d055e5-7791-4f91-a52d-314d13289239\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " Nov 03 20:32:10.814153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.814136 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8d055e5-7791-4f91-a52d-314d13289239-kube-api-access-xv6hx\" (UniqueName: \"kubernetes.io/projected/e8d055e5-7791-4f91-a52d-314d13289239-kube-api-access-xv6hx\") pod \"e8d055e5-7791-4f91-a52d-314d13289239\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " Nov 03 20:32:10.814221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.814191 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d055e5-7791-4f91-a52d-314d13289239-openshift-service-ca" pod "e8d055e5-7791-4f91-a52d-314d13289239" (UID: "e8d055e5-7791-4f91-a52d-314d13289239"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:10.814288 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.814268 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-metrics-certs\") pod \"e8d055e5-7791-4f91-a52d-314d13289239\" (UID: \"e8d055e5-7791-4f91-a52d-314d13289239\") " Nov 03 20:32:10.815894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.815236 4457 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8d055e5-7791-4f91-a52d-314d13289239-openshift-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:10.816920 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.816898 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d055e5-7791-4f91-a52d-314d13289239-kube-api-access-xv6hx" pod "e8d055e5-7791-4f91-a52d-314d13289239" (UID: "e8d055e5-7791-4f91-a52d-314d13289239"). InnerVolumeSpecName "kube-api-access-xv6hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:10.818351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.818302 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-metrics-certs" pod "e8d055e5-7791-4f91-a52d-314d13289239" (UID: "e8d055e5-7791-4f91-a52d-314d13289239"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:10.822979 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.822952 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-maas-api-tls" pod "e8d055e5-7791-4f91-a52d-314d13289239" (UID: "e8d055e5-7791-4f91-a52d-314d13289239"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:10.918532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.918235 4457 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-metrics-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:10.918532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.918271 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e8d055e5-7791-4f91-a52d-314d13289239-maas-api-tls\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:10.918532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:10.918285 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv6hx\" (UniqueName: \"kubernetes.io/projected/e8d055e5-7791-4f91-a52d-314d13289239-kube-api-access-xv6hx\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:11.572787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.572743 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" event={"ID":"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1","Type":"ContainerStarted","Data":"781eb79cc62759db81bea5a3c36a5e0f567abdd780e111da0302553b909125ad"} Nov 03 20:32:11.573016 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.572823 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" event={"ID":"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1","Type":"ContainerStarted","Data":"6d40b20c058340b4ad73eb12afb1e4127d424ebb0e73c65d97252bb394306b93"} Nov 03 20:32:11.575177 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.575150 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" Nov 03 20:32:11.575327 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.575145 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5" event={"ID":"e8d055e5-7791-4f91-a52d-314d13289239","Type":"ContainerDied","Data":"726b859e045dfddf16b6c0a6be918166b3c0ef6e3b71628577ec8ea9f658fbf9"} Nov 03 20:32:11.575327 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.575279 4457 scope.go:122] "RemoveContainer" containerID="88883fc000b94f46e32228a4017f26c61fa9138cae7fcda5b4b5f88ec8593a6f" Nov 03 20:32:11.601308 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.601259 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" podStartSLOduration=2.601244153 podStartE2EDuration="2.601244153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:11.595717154 +0000 UTC m=+2109.971313901" watchObservedRunningTime="2026-11-03 20:32:11.601244153 +0000 UTC m=+2109.976840900" Nov 03 20:32:11.619006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.618975 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5"] Nov 03 20:32:11.624745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.624712 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-77e83b-7f6df85f8-qvml5"] Nov 03 20:32:11.629589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.629562 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72"] Nov 03 20:32:11.630462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.630444 4457 cpu_manager.go:414] "containerMap: removing container" podUID="e8d055e5-7791-4f91-a52d-314d13289239" containerName="maas-api" Nov 03 20:32:11.630527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.630463 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8d055e5-7791-4f91-a52d-314d13289239" containerName="maas-api" Nov 03 20:32:11.630629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.630620 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8d055e5-7791-4f91-a52d-314d13289239" containerName="maas-api" Nov 03 20:32:11.636192 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.636169 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" Nov 03 20:32:11.641887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.641857 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72"] Nov 03 20:32:11.741464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.741417 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383998a9-gbc72\" (UID: \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" Nov 03 20:32:11.741713 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.741690 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383998a9-gbc72\" (UID: \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" Nov 03 20:32:11.741845 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.741740 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmn85\" (UniqueName: \"kubernetes.io/projected/4679afb7-c483-44e4-9ae9-c6da4dffbd44-kube-api-access-mmn85\") pod \"maas-api-revoke-sub-383998a9-gbc72\" (UID: \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" Nov 03 20:32:11.762341 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.762308 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="923c18ff-2848-495b-836a-ae922f1880d4" path="/var/lib/kubelet/pods/923c18ff-2848-495b-836a-ae922f1880d4/volumes" Nov 03 20:32:11.762962 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.762942 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8d055e5-7791-4f91-a52d-314d13289239" path="/var/lib/kubelet/pods/e8d055e5-7791-4f91-a52d-314d13289239/volumes" Nov 03 20:32:11.844072 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.843976 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmn85\" (UniqueName: \"kubernetes.io/projected/4679afb7-c483-44e4-9ae9-c6da4dffbd44-kube-api-access-mmn85\") pod \"maas-api-revoke-sub-383998a9-gbc72\" (UID: \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" Nov 03 20:32:11.844264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.844221 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383998a9-gbc72\" (UID: \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" Nov 03 20:32:11.844522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.844495 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383998a9-gbc72\" (UID: \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" Nov 03 20:32:11.845277 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.845247 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383998a9-gbc72\" (UID: \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" Nov 03 20:32:11.847103 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.847077 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383998a9-gbc72\" (UID: \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" Nov 03 20:32:11.856910 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.856884 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmn85\" (UniqueName: \"kubernetes.io/projected/4679afb7-c483-44e4-9ae9-c6da4dffbd44-kube-api-access-mmn85\") pod \"maas-api-revoke-sub-383998a9-gbc72\" (UID: \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" Nov 03 20:32:11.959497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:11.959464 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" Nov 03 20:32:12.254594 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:32:12.254562 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4679afb7_c483_44e4_9ae9_c6da4dffbd44.slice/crio-fa416775b41548b21fd05f5d1a8b5dd2fa558736c50680c3f41fb508fdd588cc WatchSource:0}: Error finding container fa416775b41548b21fd05f5d1a8b5dd2fa558736c50680c3f41fb508fdd588cc: Status 404 returned error can't find the container with id fa416775b41548b21fd05f5d1a8b5dd2fa558736c50680c3f41fb508fdd588cc Nov 03 20:32:12.254790 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.254765 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72"] Nov 03 20:32:12.302326 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.302220 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx"] Nov 03 20:32:12.309104 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.309076 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" Nov 03 20:32:12.315442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.315412 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx"] Nov 03 20:32:12.457709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.457649 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d17e52e-d2nwx\" (UID: \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" Nov 03 20:32:12.458299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.458015 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d17e52e-d2nwx\" (UID: \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" Nov 03 20:32:12.458299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.458152 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl56g\" (UniqueName: \"kubernetes.io/projected/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-kube-api-access-sl56g\") pod \"maas-api-revoke-sub-0d17e52e-d2nwx\" (UID: \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" Nov 03 20:32:12.559892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.559857 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d17e52e-d2nwx\" (UID: \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" Nov 03 20:32:12.560050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.559956 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl56g\" (UniqueName: \"kubernetes.io/projected/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-kube-api-access-sl56g\") pod \"maas-api-revoke-sub-0d17e52e-d2nwx\" (UID: \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" Nov 03 20:32:12.560146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.560062 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d17e52e-d2nwx\" (UID: \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" Nov 03 20:32:12.560747 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.560717 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d17e52e-d2nwx\" (UID: \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" Nov 03 20:32:12.562282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.562264 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d17e52e-d2nwx\" (UID: \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" Nov 03 20:32:12.572832 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.572808 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl56g\" (UniqueName: \"kubernetes.io/projected/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-kube-api-access-sl56g\") pod \"maas-api-revoke-sub-0d17e52e-d2nwx\" (UID: \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" Nov 03 20:32:12.586872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.586838 4457 generic.go:356] "Generic (PLEG): container finished" podID="d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1" containerID="781eb79cc62759db81bea5a3c36a5e0f567abdd780e111da0302553b909125ad" exitCode=0 Nov 03 20:32:12.587003 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.586915 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" event={"ID":"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1","Type":"ContainerDied","Data":"781eb79cc62759db81bea5a3c36a5e0f567abdd780e111da0302553b909125ad"} Nov 03 20:32:12.589538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.589512 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" event={"ID":"4679afb7-c483-44e4-9ae9-c6da4dffbd44","Type":"ContainerStarted","Data":"fa416775b41548b21fd05f5d1a8b5dd2fa558736c50680c3f41fb508fdd588cc"} Nov 03 20:32:12.640201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.640171 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" Nov 03 20:32:12.936623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:12.936593 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx"] Nov 03 20:32:13.026245 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.026208 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" podUID="9e5dbdf1-f78c-4368-aabb-232578d293a7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:32:13.026650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.026306 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" podUID="130c5e5b-43ee-4329-ab13-f37b6c3981fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:32:13.601920 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.601880 4457 generic.go:356] "Generic (PLEG): container finished" podID="4679afb7-c483-44e4-9ae9-c6da4dffbd44" containerID="31c8e43d3ab3a6b4944ddcffc56fe287c05804f6d902e9e69f293c37edcc01fa" exitCode=0 Nov 03 20:32:13.602099 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.601962 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" event={"ID":"4679afb7-c483-44e4-9ae9-c6da4dffbd44","Type":"ContainerDied","Data":"31c8e43d3ab3a6b4944ddcffc56fe287c05804f6d902e9e69f293c37edcc01fa"} Nov 03 20:32:13.604093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.604066 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" event={"ID":"3b9323b3-cc79-4b20-a4fb-d69e42e0e038","Type":"ContainerStarted","Data":"1ec8e144f67ae9c68c5f9e94dacccec85c72c984779f0dcef154290db59af9c4"} Nov 03 20:32:13.604214 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.604122 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" event={"ID":"3b9323b3-cc79-4b20-a4fb-d69e42e0e038","Type":"ContainerStarted","Data":"45757097d9851ed781b299999af3bdbf324711f459379dfed86ac7cfbbcbaf52"} Nov 03 20:32:13.647840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.647729 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" podStartSLOduration=1.647714634 podStartE2EDuration="1.647714634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:13.645398259 +0000 UTC m=+2112.020994999" watchObservedRunningTime="2026-11-03 20:32:13.647714634 +0000 UTC m=+2112.023311381" Nov 03 20:32:13.798099 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.798068 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" Nov 03 20:32:13.882824 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.882792 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-kube-api-access-c76md\" (UniqueName: \"kubernetes.io/projected/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-kube-api-access-c76md\") pod \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\" (UID: \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\") " Nov 03 20:32:13.883033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.882900 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-xks-ca\") pod \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\" (UID: \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\") " Nov 03 20:32:13.883033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.883028 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-openshift-ca\") pod \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\" (UID: \"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1\") " Nov 03 20:32:13.883511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.883469 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-openshift-ca" pod "d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1" (UID: "d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:13.884366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.884339 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:13.885272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.885251 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-kube-api-access-c76md" pod "d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1" (UID: "d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1"). InnerVolumeSpecName "kube-api-access-c76md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:13.885467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.885440 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-xks-ca" pod "d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1" (UID: "d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:13.986471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.986345 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c76md\" (UniqueName: \"kubernetes.io/projected/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-kube-api-access-c76md\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:13.986471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:13.986396 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:14.615692 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:14.615665 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" Nov 03 20:32:14.616131 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:14.615709 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2" event={"ID":"d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1","Type":"ContainerDied","Data":"6d40b20c058340b4ad73eb12afb1e4127d424ebb0e73c65d97252bb394306b93"} Nov 03 20:32:14.616131 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:14.615757 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d40b20c058340b4ad73eb12afb1e4127d424ebb0e73c65d97252bb394306b93" Nov 03 20:32:14.618014 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:14.617983 4457 generic.go:356] "Generic (PLEG): container finished" podID="3b9323b3-cc79-4b20-a4fb-d69e42e0e038" containerID="1ec8e144f67ae9c68c5f9e94dacccec85c72c984779f0dcef154290db59af9c4" exitCode=0 Nov 03 20:32:14.618146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:14.618072 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" event={"ID":"3b9323b3-cc79-4b20-a4fb-d69e42e0e038","Type":"ContainerDied","Data":"1ec8e144f67ae9c68c5f9e94dacccec85c72c984779f0dcef154290db59af9c4"} Nov 03 20:32:14.870126 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:14.870055 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" Nov 03 20:32:15.007470 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.007438 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4679afb7-c483-44e4-9ae9-c6da4dffbd44-kube-api-access-mmn85\" (UniqueName: \"kubernetes.io/projected/4679afb7-c483-44e4-9ae9-c6da4dffbd44-kube-api-access-mmn85\") pod \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\" (UID: \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\") " Nov 03 20:32:15.007638 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.007536 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-openshift-ca\") pod \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\" (UID: \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\") " Nov 03 20:32:15.007638 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.007602 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-xks-ca\") pod \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\" (UID: \"4679afb7-c483-44e4-9ae9-c6da4dffbd44\") " Nov 03 20:32:15.007952 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.007927 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-openshift-ca" pod "4679afb7-c483-44e4-9ae9-c6da4dffbd44" (UID: "4679afb7-c483-44e4-9ae9-c6da4dffbd44"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:15.008452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.008381 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:15.009835 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.009811 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-xks-ca" pod "4679afb7-c483-44e4-9ae9-c6da4dffbd44" (UID: "4679afb7-c483-44e4-9ae9-c6da4dffbd44"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:15.009968 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.009944 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4679afb7-c483-44e4-9ae9-c6da4dffbd44-kube-api-access-mmn85" pod "4679afb7-c483-44e4-9ae9-c6da4dffbd44" (UID: "4679afb7-c483-44e4-9ae9-c6da4dffbd44"). InnerVolumeSpecName "kube-api-access-mmn85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:15.110211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.110171 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmn85\" (UniqueName: \"kubernetes.io/projected/4679afb7-c483-44e4-9ae9-c6da4dffbd44-kube-api-access-mmn85\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:15.110211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.110211 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4679afb7-c483-44e4-9ae9-c6da4dffbd44-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:15.628898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.628863 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" Nov 03 20:32:15.628898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.628875 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72" event={"ID":"4679afb7-c483-44e4-9ae9-c6da4dffbd44","Type":"ContainerDied","Data":"fa416775b41548b21fd05f5d1a8b5dd2fa558736c50680c3f41fb508fdd588cc"} Nov 03 20:32:15.629358 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.628949 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa416775b41548b21fd05f5d1a8b5dd2fa558736c50680c3f41fb508fdd588cc" Nov 03 20:32:15.834493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.834467 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" Nov 03 20:32:15.927612 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.927515 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-openshift-ca\") pod \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\" (UID: \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\") " Nov 03 20:32:15.927782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.927613 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-kube-api-access-sl56g\" (UniqueName: \"kubernetes.io/projected/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-kube-api-access-sl56g\") pod \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\" (UID: \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\") " Nov 03 20:32:15.927782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.927680 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-xks-ca\") pod \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\" (UID: \"3b9323b3-cc79-4b20-a4fb-d69e42e0e038\") " Nov 03 20:32:15.927991 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.927962 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-openshift-ca" pod "3b9323b3-cc79-4b20-a4fb-d69e42e0e038" (UID: "3b9323b3-cc79-4b20-a4fb-d69e42e0e038"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:15.928621 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.928590 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:15.930604 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.930578 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-xks-ca" pod "3b9323b3-cc79-4b20-a4fb-d69e42e0e038" (UID: "3b9323b3-cc79-4b20-a4fb-d69e42e0e038"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:15.930839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:15.930748 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-kube-api-access-sl56g" pod "3b9323b3-cc79-4b20-a4fb-d69e42e0e038" (UID: "3b9323b3-cc79-4b20-a4fb-d69e42e0e038"). InnerVolumeSpecName "kube-api-access-sl56g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:16.030497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:16.030462 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl56g\" (UniqueName: \"kubernetes.io/projected/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-kube-api-access-sl56g\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:16.030497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:16.030492 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b9323b3-cc79-4b20-a4fb-d69e42e0e038-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:16.639697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:16.639659 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" Nov 03 20:32:16.639697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:16.639687 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx" event={"ID":"3b9323b3-cc79-4b20-a4fb-d69e42e0e038","Type":"ContainerDied","Data":"45757097d9851ed781b299999af3bdbf324711f459379dfed86ac7cfbbcbaf52"} Nov 03 20:32:16.640125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:16.639738 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45757097d9851ed781b299999af3bdbf324711f459379dfed86ac7cfbbcbaf52" Nov 03 20:32:17.592990 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.592960 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" Nov 03 20:32:17.597987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.597961 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" Nov 03 20:32:17.661800 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.661583 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5dbdf1-f78c-4368-aabb-232578d293a7-kube-api-access-ng49b\" (UniqueName: \"kubernetes.io/projected/9e5dbdf1-f78c-4368-aabb-232578d293a7-kube-api-access-ng49b\") pod \"9e5dbdf1-f78c-4368-aabb-232578d293a7\" (UID: \"9e5dbdf1-f78c-4368-aabb-232578d293a7\") " Nov 03 20:32:17.661800 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.661803 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/130c5e5b-43ee-4329-ab13-f37b6c3981fb-kube-api-access-c5kmw\" (UniqueName: \"kubernetes.io/projected/130c5e5b-43ee-4329-ab13-f37b6c3981fb-kube-api-access-c5kmw\") pod \"130c5e5b-43ee-4329-ab13-f37b6c3981fb\" (UID: \"130c5e5b-43ee-4329-ab13-f37b6c3981fb\") " Nov 03 20:32:17.662374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.661844 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/130c5e5b-43ee-4329-ab13-f37b6c3981fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130c5e5b-43ee-4329-ab13-f37b6c3981fb-plugins-config-volume\") pod \"130c5e5b-43ee-4329-ab13-f37b6c3981fb\" (UID: \"130c5e5b-43ee-4329-ab13-f37b6c3981fb\") " Nov 03 20:32:17.662374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.661911 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5dbdf1-f78c-4368-aabb-232578d293a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5dbdf1-f78c-4368-aabb-232578d293a7-plugins-config-volume\") pod \"9e5dbdf1-f78c-4368-aabb-232578d293a7\" (UID: \"9e5dbdf1-f78c-4368-aabb-232578d293a7\") " Nov 03 20:32:17.662868 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.662742 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5dbdf1-f78c-4368-aabb-232578d293a7-plugins-config-volume" pod "9e5dbdf1-f78c-4368-aabb-232578d293a7" (UID: "9e5dbdf1-f78c-4368-aabb-232578d293a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:17.663205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.663180 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130c5e5b-43ee-4329-ab13-f37b6c3981fb-plugins-config-volume" pod "130c5e5b-43ee-4329-ab13-f37b6c3981fb" (UID: "130c5e5b-43ee-4329-ab13-f37b6c3981fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:17.663636 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.663613 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" Nov 03 20:32:17.663742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.663660 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" event={"ID":"9e5dbdf1-f78c-4368-aabb-232578d293a7","Type":"ContainerDied","Data":"de36493ec97096c2941d816c9d0c5b2905711a7f834f593a9a27a7bde209baaf"} Nov 03 20:32:17.663811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.663750 4457 scope.go:122] "RemoveContainer" containerID="de36493ec97096c2941d816c9d0c5b2905711a7f834f593a9a27a7bde209baaf" Nov 03 20:32:17.664526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.663535 4457 generic.go:356] "Generic (PLEG): container finished" podID="9e5dbdf1-f78c-4368-aabb-232578d293a7" containerID="de36493ec97096c2941d816c9d0c5b2905711a7f834f593a9a27a7bde209baaf" exitCode=0 Nov 03 20:32:17.664651 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.664630 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx" event={"ID":"9e5dbdf1-f78c-4368-aabb-232578d293a7","Type":"ContainerDied","Data":"ceb4c2d5499a364aaf582d7f179cd4dbc5ebe9b90f71fe4f77d826443b4ba12f"} Nov 03 20:32:17.665873 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.665840 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130c5e5b-43ee-4329-ab13-f37b6c3981fb-kube-api-access-c5kmw" pod "130c5e5b-43ee-4329-ab13-f37b6c3981fb" (UID: "130c5e5b-43ee-4329-ab13-f37b6c3981fb"). InnerVolumeSpecName "kube-api-access-c5kmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:17.666834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.666752 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5dbdf1-f78c-4368-aabb-232578d293a7-kube-api-access-ng49b" pod "9e5dbdf1-f78c-4368-aabb-232578d293a7" (UID: "9e5dbdf1-f78c-4368-aabb-232578d293a7"). InnerVolumeSpecName "kube-api-access-ng49b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:17.667440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.667283 4457 generic.go:356] "Generic (PLEG): container finished" podID="130c5e5b-43ee-4329-ab13-f37b6c3981fb" containerID="b3c4f49d657b41cbb403533d2a66b6d862a14b489768801f0c1b628b0fecc44f" exitCode=0 Nov 03 20:32:17.667646 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.667618 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" Nov 03 20:32:17.668145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.668120 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" event={"ID":"130c5e5b-43ee-4329-ab13-f37b6c3981fb","Type":"ContainerDied","Data":"b3c4f49d657b41cbb403533d2a66b6d862a14b489768801f0c1b628b0fecc44f"} Nov 03 20:32:17.668226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.668174 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl" event={"ID":"130c5e5b-43ee-4329-ab13-f37b6c3981fb","Type":"ContainerDied","Data":"5c3bc138ea6a29d30469b7c88a1229f3e3d0cd08d16d4e27d4a12937daafce11"} Nov 03 20:32:17.683622 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.683601 4457 scope.go:122] "RemoveContainer" containerID="de36493ec97096c2941d816c9d0c5b2905711a7f834f593a9a27a7bde209baaf" Nov 03 20:32:17.684007 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:32:17.683980 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de36493ec97096c2941d816c9d0c5b2905711a7f834f593a9a27a7bde209baaf\": container with ID starting with de36493ec97096c2941d816c9d0c5b2905711a7f834f593a9a27a7bde209baaf not found: ID does not exist" containerID="de36493ec97096c2941d816c9d0c5b2905711a7f834f593a9a27a7bde209baaf" Nov 03 20:32:17.684150 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.684018 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de36493ec97096c2941d816c9d0c5b2905711a7f834f593a9a27a7bde209baaf"} err="failed to get container status \"de36493ec97096c2941d816c9d0c5b2905711a7f834f593a9a27a7bde209baaf\": rpc error: code = NotFound desc = could not find container \"de36493ec97096c2941d816c9d0c5b2905711a7f834f593a9a27a7bde209baaf\": container with ID starting with de36493ec97096c2941d816c9d0c5b2905711a7f834f593a9a27a7bde209baaf not found: ID does not exist" Nov 03 20:32:17.684150 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.684044 4457 scope.go:122] "RemoveContainer" containerID="b3c4f49d657b41cbb403533d2a66b6d862a14b489768801f0c1b628b0fecc44f" Nov 03 20:32:17.743089 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.743052 4457 scope.go:122] "RemoveContainer" containerID="b3c4f49d657b41cbb403533d2a66b6d862a14b489768801f0c1b628b0fecc44f" Nov 03 20:32:17.743089 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:32:17.743528 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3c4f49d657b41cbb403533d2a66b6d862a14b489768801f0c1b628b0fecc44f\": container with ID starting with b3c4f49d657b41cbb403533d2a66b6d862a14b489768801f0c1b628b0fecc44f not found: ID does not exist" containerID="b3c4f49d657b41cbb403533d2a66b6d862a14b489768801f0c1b628b0fecc44f" Nov 03 20:32:17.743089 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.743570 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3c4f49d657b41cbb403533d2a66b6d862a14b489768801f0c1b628b0fecc44f"} err="failed to get container status \"b3c4f49d657b41cbb403533d2a66b6d862a14b489768801f0c1b628b0fecc44f\": rpc error: code = NotFound desc = could not find container \"b3c4f49d657b41cbb403533d2a66b6d862a14b489768801f0c1b628b0fecc44f\": container with ID starting with b3c4f49d657b41cbb403533d2a66b6d862a14b489768801f0c1b628b0fecc44f not found: ID does not exist" Nov 03 20:32:17.746206 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.746154 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl"] Nov 03 20:32:17.766012 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.765980 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e5dbdf1-f78c-4368-aabb-232578d293a7-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:17.766264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.766237 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng49b\" (UniqueName: \"kubernetes.io/projected/9e5dbdf1-f78c-4368-aabb-232578d293a7-kube-api-access-ng49b\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:17.766487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.766466 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5kmw\" (UniqueName: \"kubernetes.io/projected/130c5e5b-43ee-4329-ab13-f37b6c3981fb-kube-api-access-c5kmw\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:17.766651 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.766630 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/130c5e5b-43ee-4329-ab13-f37b6c3981fb-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:17.805347 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:17.805313 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-77e83b-c9c588f7b-866jl"] Nov 03 20:32:18.015704 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.015674 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx"] Nov 03 20:32:18.043217 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.043182 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-77e83b-6bfc65cff5-qwzsx"] Nov 03 20:32:18.058173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.058086 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw"] Nov 03 20:32:18.058429 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.058358 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" podUID="32389e55-4f71-49b0-ba54-d53d241843b1" containerName="istio-proxy" containerID="cri-o://318b199b14fa9e62ef7bd6af9bd6e35f916aba5d5806ab14da31e176a139813a" gracePeriod=30 Nov 03 20:32:18.505833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.505747 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k"] Nov 03 20:32:18.506913 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.506882 4457 cpu_manager.go:414] "containerMap: removing container" podUID="3b9323b3-cc79-4b20-a4fb-d69e42e0e038" containerName="revoke-keys" Nov 03 20:32:18.506913 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.506911 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b9323b3-cc79-4b20-a4fb-d69e42e0e038" containerName="revoke-keys" Nov 03 20:32:18.507120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.506956 4457 cpu_manager.go:414] "containerMap: removing container" podUID="9e5dbdf1-f78c-4368-aabb-232578d293a7" containerName="payload-pre-processing" Nov 03 20:32:18.507120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.506964 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5dbdf1-f78c-4368-aabb-232578d293a7" containerName="payload-pre-processing" Nov 03 20:32:18.507120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.507011 4457 cpu_manager.go:414] "containerMap: removing container" podUID="130c5e5b-43ee-4329-ab13-f37b6c3981fb" containerName="payload-processing" Nov 03 20:32:18.507120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.507019 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130c5e5b-43ee-4329-ab13-f37b6c3981fb" containerName="payload-processing" Nov 03 20:32:18.507120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.507041 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1" containerName="revoke-keys" Nov 03 20:32:18.507120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.507048 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1" containerName="revoke-keys" Nov 03 20:32:18.507120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.507067 4457 cpu_manager.go:414] "containerMap: removing container" podUID="4679afb7-c483-44e4-9ae9-c6da4dffbd44" containerName="revoke-keys" Nov 03 20:32:18.507120 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.507074 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4679afb7-c483-44e4-9ae9-c6da4dffbd44" containerName="revoke-keys" Nov 03 20:32:18.507570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.507267 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b9323b3-cc79-4b20-a4fb-d69e42e0e038" containerName="revoke-keys" Nov 03 20:32:18.507570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.507290 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5dbdf1-f78c-4368-aabb-232578d293a7" containerName="payload-pre-processing" Nov 03 20:32:18.507570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.507308 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="130c5e5b-43ee-4329-ab13-f37b6c3981fb" containerName="payload-processing" Nov 03 20:32:18.507570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.507320 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1" containerName="revoke-keys" Nov 03 20:32:18.507570 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.507330 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="4679afb7-c483-44e4-9ae9-c6da4dffbd44" containerName="revoke-keys" Nov 03 20:32:18.513123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.513093 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" Nov 03 20:32:18.518446 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.518414 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k"] Nov 03 20:32:18.593487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.592843 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38264330-3138-4328-95a7-de830575c54f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34e844-2s27k\" (UID: \"38264330-3138-4328-95a7-de830575c54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" Nov 03 20:32:18.593487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.593017 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rltn\" (UniqueName: \"kubernetes.io/projected/38264330-3138-4328-95a7-de830575c54f-kube-api-access-8rltn\") pod \"maas-api-revoke-keys-e2e-inf-b-34e844-2s27k\" (UID: \"38264330-3138-4328-95a7-de830575c54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" Nov 03 20:32:18.593487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.593134 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38264330-3138-4328-95a7-de830575c54f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34e844-2s27k\" (UID: \"38264330-3138-4328-95a7-de830575c54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" Nov 03 20:32:18.604901 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.604874 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:32:18.605076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.604967 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:32:18.698083 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.698015 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38264330-3138-4328-95a7-de830575c54f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34e844-2s27k\" (UID: \"38264330-3138-4328-95a7-de830575c54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" Nov 03 20:32:18.698624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.698193 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rltn\" (UniqueName: \"kubernetes.io/projected/38264330-3138-4328-95a7-de830575c54f-kube-api-access-8rltn\") pod \"maas-api-revoke-keys-e2e-inf-b-34e844-2s27k\" (UID: \"38264330-3138-4328-95a7-de830575c54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" Nov 03 20:32:18.698624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.698284 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38264330-3138-4328-95a7-de830575c54f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34e844-2s27k\" (UID: \"38264330-3138-4328-95a7-de830575c54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" Nov 03 20:32:18.699782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.699751 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38264330-3138-4328-95a7-de830575c54f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34e844-2s27k\" (UID: \"38264330-3138-4328-95a7-de830575c54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" Nov 03 20:32:18.706397 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.706355 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38264330-3138-4328-95a7-de830575c54f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-34e844-2s27k\" (UID: \"38264330-3138-4328-95a7-de830575c54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" Nov 03 20:32:18.712381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.712350 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rltn\" (UniqueName: \"kubernetes.io/projected/38264330-3138-4328-95a7-de830575c54f-kube-api-access-8rltn\") pod \"maas-api-revoke-keys-e2e-inf-b-34e844-2s27k\" (UID: \"38264330-3138-4328-95a7-de830575c54f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" Nov 03 20:32:18.838443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:18.838375 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" Nov 03 20:32:19.139472 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:32:19.139434 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38264330_3138_4328_95a7_de830575c54f.slice/crio-9d16312b74bf0d2342c868bec4c44f64bcb23f156a2a883ec76e5de62bf09ab8 WatchSource:0}: Error finding container 9d16312b74bf0d2342c868bec4c44f64bcb23f156a2a883ec76e5de62bf09ab8: Status 404 returned error can't find the container with id 9d16312b74bf0d2342c868bec4c44f64bcb23f156a2a883ec76e5de62bf09ab8 Nov 03 20:32:19.139882 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:19.139858 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k"] Nov 03 20:32:19.706363 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:19.706325 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" event={"ID":"38264330-3138-4328-95a7-de830575c54f","Type":"ContainerStarted","Data":"a241caeecce6275f5facfd52904965f33032c9d153cd13273602a57cfaab6d65"} Nov 03 20:32:19.706805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:19.706426 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" event={"ID":"38264330-3138-4328-95a7-de830575c54f","Type":"ContainerStarted","Data":"9d16312b74bf0d2342c868bec4c44f64bcb23f156a2a883ec76e5de62bf09ab8"} Nov 03 20:32:19.714190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:19.714161 4457 patch_prober.go:29] interesting pod/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:32:19.714344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:19.714226 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" podUID="32389e55-4f71-49b0-ba54-d53d241843b1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:32:19.726742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:19.726692 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" podStartSLOduration=1.726678636 podStartE2EDuration="1.726678636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:19.724268413 +0000 UTC m=+2118.099865185" watchObservedRunningTime="2026-11-03 20:32:19.726678636 +0000 UTC m=+2118.102275383" Nov 03 20:32:19.764276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:19.764242 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="130c5e5b-43ee-4329-ab13-f37b6c3981fb" path="/var/lib/kubelet/pods/130c5e5b-43ee-4329-ab13-f37b6c3981fb/volumes" Nov 03 20:32:19.764835 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:19.764818 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5dbdf1-f78c-4368-aabb-232578d293a7" path="/var/lib/kubelet/pods/9e5dbdf1-f78c-4368-aabb-232578d293a7/volumes" Nov 03 20:32:20.722787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:20.722743 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" event={"ID":"38264330-3138-4328-95a7-de830575c54f","Type":"ContainerDied","Data":"a241caeecce6275f5facfd52904965f33032c9d153cd13273602a57cfaab6d65"} Nov 03 20:32:20.723293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:20.722952 4457 generic.go:356] "Generic (PLEG): container finished" podID="38264330-3138-4328-95a7-de830575c54f" containerID="a241caeecce6275f5facfd52904965f33032c9d153cd13273602a57cfaab6d65" exitCode=0 Nov 03 20:32:20.888488 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:20.888452 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w"] Nov 03 20:32:20.932048 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:20.930435 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w"] Nov 03 20:32:20.932048 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:20.930562 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" Nov 03 20:32:21.072123 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:21.072081 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a7e526-m9j2w\" (UID: \"396c4ede-c1f9-4a2b-8130-17144013bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" Nov 03 20:32:21.072311 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:21.072281 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2g77\" (UniqueName: \"kubernetes.io/projected/396c4ede-c1f9-4a2b-8130-17144013bf31-kube-api-access-z2g77\") pod \"maas-api-revoke-sub-65a7e526-m9j2w\" (UID: \"396c4ede-c1f9-4a2b-8130-17144013bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" Nov 03 20:32:21.072551 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:21.072524 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a7e526-m9j2w\" (UID: \"396c4ede-c1f9-4a2b-8130-17144013bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" Nov 03 20:32:21.174674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:21.174637 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a7e526-m9j2w\" (UID: \"396c4ede-c1f9-4a2b-8130-17144013bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" Nov 03 20:32:21.174851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:21.174773 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a7e526-m9j2w\" (UID: \"396c4ede-c1f9-4a2b-8130-17144013bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" Nov 03 20:32:21.174949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:21.174932 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2g77\" (UniqueName: \"kubernetes.io/projected/396c4ede-c1f9-4a2b-8130-17144013bf31-kube-api-access-z2g77\") pod \"maas-api-revoke-sub-65a7e526-m9j2w\" (UID: \"396c4ede-c1f9-4a2b-8130-17144013bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" Nov 03 20:32:21.175377 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:21.175351 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65a7e526-m9j2w\" (UID: \"396c4ede-c1f9-4a2b-8130-17144013bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" Nov 03 20:32:21.177623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:21.177595 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65a7e526-m9j2w\" (UID: \"396c4ede-c1f9-4a2b-8130-17144013bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" Nov 03 20:32:21.192047 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:21.192014 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2g77\" (UniqueName: \"kubernetes.io/projected/396c4ede-c1f9-4a2b-8130-17144013bf31-kube-api-access-z2g77\") pod \"maas-api-revoke-sub-65a7e526-m9j2w\" (UID: \"396c4ede-c1f9-4a2b-8130-17144013bf31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" Nov 03 20:32:21.263328 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:21.263286 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" Nov 03 20:32:21.556523 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:32:21.556489 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396c4ede_c1f9_4a2b_8130_17144013bf31.slice/crio-d708b79c69a48e78d78ed97b9c4416449b78335b12c314208e5785ef4e040a56 WatchSource:0}: Error finding container d708b79c69a48e78d78ed97b9c4416449b78335b12c314208e5785ef4e040a56: Status 404 returned error can't find the container with id d708b79c69a48e78d78ed97b9c4416449b78335b12c314208e5785ef4e040a56 Nov 03 20:32:21.557683 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:21.557601 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w"] Nov 03 20:32:21.735711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:21.735677 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" event={"ID":"396c4ede-c1f9-4a2b-8130-17144013bf31","Type":"ContainerStarted","Data":"d708b79c69a48e78d78ed97b9c4416449b78335b12c314208e5785ef4e040a56"} Nov 03 20:32:21.939955 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:21.939920 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" Nov 03 20:32:22.095792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.095760 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38264330-3138-4328-95a7-de830575c54f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38264330-3138-4328-95a7-de830575c54f-maas-api-openshift-ca\") pod \"38264330-3138-4328-95a7-de830575c54f\" (UID: \"38264330-3138-4328-95a7-de830575c54f\") " Nov 03 20:32:22.095943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.095865 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38264330-3138-4328-95a7-de830575c54f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38264330-3138-4328-95a7-de830575c54f-maas-api-xks-ca\") pod \"38264330-3138-4328-95a7-de830575c54f\" (UID: \"38264330-3138-4328-95a7-de830575c54f\") " Nov 03 20:32:22.096024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.095940 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38264330-3138-4328-95a7-de830575c54f-kube-api-access-8rltn\" (UniqueName: \"kubernetes.io/projected/38264330-3138-4328-95a7-de830575c54f-kube-api-access-8rltn\") pod \"38264330-3138-4328-95a7-de830575c54f\" (UID: \"38264330-3138-4328-95a7-de830575c54f\") " Nov 03 20:32:22.096226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.096185 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38264330-3138-4328-95a7-de830575c54f-maas-api-openshift-ca" pod "38264330-3138-4328-95a7-de830575c54f" (UID: "38264330-3138-4328-95a7-de830575c54f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:22.096891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.096867 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38264330-3138-4328-95a7-de830575c54f-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:22.098720 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.098695 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38264330-3138-4328-95a7-de830575c54f-maas-api-xks-ca" pod "38264330-3138-4328-95a7-de830575c54f" (UID: "38264330-3138-4328-95a7-de830575c54f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:22.099029 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.099005 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38264330-3138-4328-95a7-de830575c54f-kube-api-access-8rltn" pod "38264330-3138-4328-95a7-de830575c54f" (UID: "38264330-3138-4328-95a7-de830575c54f"). InnerVolumeSpecName "kube-api-access-8rltn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:22.198959 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.198868 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rltn\" (UniqueName: \"kubernetes.io/projected/38264330-3138-4328-95a7-de830575c54f-kube-api-access-8rltn\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:22.198959 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.198907 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38264330-3138-4328-95a7-de830575c54f-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:22.747887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.747846 4457 generic.go:356] "Generic (PLEG): container finished" podID="396c4ede-c1f9-4a2b-8130-17144013bf31" containerID="711c7cd70db2594ed94782f82cc81379e19d5c82ed8a456a852ae817e1b12150" exitCode=0 Nov 03 20:32:22.748359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.747930 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" event={"ID":"396c4ede-c1f9-4a2b-8130-17144013bf31","Type":"ContainerDied","Data":"711c7cd70db2594ed94782f82cc81379e19d5c82ed8a456a852ae817e1b12150"} Nov 03 20:32:22.749812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.749786 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" Nov 03 20:32:22.750302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.749839 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k" event={"ID":"38264330-3138-4328-95a7-de830575c54f","Type":"ContainerDied","Data":"9d16312b74bf0d2342c868bec4c44f64bcb23f156a2a883ec76e5de62bf09ab8"} Nov 03 20:32:22.750302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:22.749916 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d16312b74bf0d2342c868bec4c44f64bcb23f156a2a883ec76e5de62bf09ab8" Nov 03 20:32:23.372202 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.372173 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:32:23.450588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.450549 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4"] Nov 03 20:32:23.609042 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.550973 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-credential-socket\") pod \"32389e55-4f71-49b0-ba54-d53d241843b1\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " Nov 03 20:32:23.609042 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.551103 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-istio-token\" (UniqueName: \"kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-istio-token\") pod \"32389e55-4f71-49b0-ba54-d53d241843b1\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " Nov 03 20:32:23.609042 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.556820 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-credential-socket" pod "32389e55-4f71-49b0-ba54-d53d241843b1" (UID: "32389e55-4f71-49b0-ba54-d53d241843b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.617427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.612955 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-envoy\") pod \"32389e55-4f71-49b0-ba54-d53d241843b1\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " Nov 03 20:32:23.617427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.613072 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-data\") pod \"32389e55-4f71-49b0-ba54-d53d241843b1\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " Nov 03 20:32:23.617427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.613199 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-socket\") pod \"32389e55-4f71-49b0-ba54-d53d241843b1\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " Nov 03 20:32:23.617427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.615122 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-socket" pod "32389e55-4f71-49b0-ba54-d53d241843b1" (UID: "32389e55-4f71-49b0-ba54-d53d241843b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.617427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.615368 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-certs" pod "32389e55-4f71-49b0-ba54-d53d241843b1" (UID: "32389e55-4f71-49b0-ba54-d53d241843b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.617427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.615477 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-data" pod "32389e55-4f71-49b0-ba54-d53d241843b1" (UID: "32389e55-4f71-49b0-ba54-d53d241843b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.617427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.615534 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-certs\") pod \"32389e55-4f71-49b0-ba54-d53d241843b1\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " Nov 03 20:32:23.617427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.615590 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32389e55-4f71-49b0-ba54-d53d241843b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32389e55-4f71-49b0-ba54-d53d241843b1-istio-podinfo\") pod \"32389e55-4f71-49b0-ba54-d53d241843b1\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " Nov 03 20:32:23.617427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.615660 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32389e55-4f71-49b0-ba54-d53d241843b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32389e55-4f71-49b0-ba54-d53d241843b1-istiod-ca-cert\") pod \"32389e55-4f71-49b0-ba54-d53d241843b1\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " Nov 03 20:32:23.617427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.615728 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-kube-api-access-gcbqf\" (UniqueName: \"kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-kube-api-access-gcbqf\") pod \"32389e55-4f71-49b0-ba54-d53d241843b1\" (UID: \"32389e55-4f71-49b0-ba54-d53d241843b1\") " Nov 03 20:32:23.617427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.617265 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32389e55-4f71-49b0-ba54-d53d241843b1-istiod-ca-cert" pod "32389e55-4f71-49b0-ba54-d53d241843b1" (UID: "32389e55-4f71-49b0-ba54-d53d241843b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:23.618950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.618598 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.618950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.618628 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.618950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.618643 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.618950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.618658 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32389e55-4f71-49b0-ba54-d53d241843b1-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.618950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.618673 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.618950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.618881 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-istio-token" pod "32389e55-4f71-49b0-ba54-d53d241843b1" (UID: "32389e55-4f71-49b0-ba54-d53d241843b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:23.652948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.652652 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-kube-api-access-gcbqf" pod "32389e55-4f71-49b0-ba54-d53d241843b1" (UID: "32389e55-4f71-49b0-ba54-d53d241843b1"). InnerVolumeSpecName "kube-api-access-gcbqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:23.672859 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.666588 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32389e55-4f71-49b0-ba54-d53d241843b1-istio-podinfo" pod "32389e55-4f71-49b0-ba54-d53d241843b1" (UID: "32389e55-4f71-49b0-ba54-d53d241843b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:23.672859 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.668721 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-envoy" pod "32389e55-4f71-49b0-ba54-d53d241843b1" (UID: "32389e55-4f71-49b0-ba54-d53d241843b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:23.722770 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.722591 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.722770 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.722641 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32389e55-4f71-49b0-ba54-d53d241843b1-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.722770 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.722657 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32389e55-4f71-49b0-ba54-d53d241843b1-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.722770 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.722675 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcbqf\" (UniqueName: \"kubernetes.io/projected/32389e55-4f71-49b0-ba54-d53d241843b1-kube-api-access-gcbqf\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:23.823810 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.823757 4457 generic.go:356] "Generic (PLEG): container finished" podID="32389e55-4f71-49b0-ba54-d53d241843b1" containerID="318b199b14fa9e62ef7bd6af9bd6e35f916aba5d5806ab14da31e176a139813a" exitCode=0 Nov 03 20:32:23.824331 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.824213 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" Nov 03 20:32:23.829134 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.829101 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" event={"ID":"32389e55-4f71-49b0-ba54-d53d241843b1","Type":"ContainerDied","Data":"318b199b14fa9e62ef7bd6af9bd6e35f916aba5d5806ab14da31e176a139813a"} Nov 03 20:32:23.829313 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.829156 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw" event={"ID":"32389e55-4f71-49b0-ba54-d53d241843b1","Type":"ContainerDied","Data":"0b49f5da8f0ea22fa6342e13cf5b0076557266e5611c8fe859c52233695605b1"} Nov 03 20:32:23.829313 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.829180 4457 scope.go:122] "RemoveContainer" containerID="318b199b14fa9e62ef7bd6af9bd6e35f916aba5d5806ab14da31e176a139813a" Nov 03 20:32:23.847666 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.847604 4457 scope.go:122] "RemoveContainer" containerID="318b199b14fa9e62ef7bd6af9bd6e35f916aba5d5806ab14da31e176a139813a" Nov 03 20:32:23.848034 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:32:23.847955 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318b199b14fa9e62ef7bd6af9bd6e35f916aba5d5806ab14da31e176a139813a\": container with ID starting with 318b199b14fa9e62ef7bd6af9bd6e35f916aba5d5806ab14da31e176a139813a not found: ID does not exist" containerID="318b199b14fa9e62ef7bd6af9bd6e35f916aba5d5806ab14da31e176a139813a" Nov 03 20:32:23.848034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.847986 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318b199b14fa9e62ef7bd6af9bd6e35f916aba5d5806ab14da31e176a139813a"} err="failed to get container status \"318b199b14fa9e62ef7bd6af9bd6e35f916aba5d5806ab14da31e176a139813a\": rpc error: code = NotFound desc = could not find container \"318b199b14fa9e62ef7bd6af9bd6e35f916aba5d5806ab14da31e176a139813a\": container with ID starting with 318b199b14fa9e62ef7bd6af9bd6e35f916aba5d5806ab14da31e176a139813a not found: ID does not exist" Nov 03 20:32:23.861934 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.861796 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw"] Nov 03 20:32:23.872100 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:23.872061 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-77e83b-openshift-default-879c5f96c-scwcw"] Nov 03 20:32:24.009086 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.009053 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm"] Nov 03 20:32:24.010633 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.010603 4457 cpu_manager.go:414] "containerMap: removing container" podUID="38264330-3138-4328-95a7-de830575c54f" containerName="revoke-keys" Nov 03 20:32:24.010633 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.010628 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38264330-3138-4328-95a7-de830575c54f" containerName="revoke-keys" Nov 03 20:32:24.010847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.010661 4457 cpu_manager.go:414] "containerMap: removing container" podUID="32389e55-4f71-49b0-ba54-d53d241843b1" containerName="istio-proxy" Nov 03 20:32:24.010847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.010667 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32389e55-4f71-49b0-ba54-d53d241843b1" containerName="istio-proxy" Nov 03 20:32:24.010847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.010807 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="32389e55-4f71-49b0-ba54-d53d241843b1" containerName="istio-proxy" Nov 03 20:32:24.010847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.010825 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="38264330-3138-4328-95a7-de830575c54f" containerName="revoke-keys" Nov 03 20:32:24.019014 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.018982 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm"] Nov 03 20:32:24.019173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.019116 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" Nov 03 20:32:24.053904 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.053871 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" Nov 03 20:32:24.129844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.129736 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396c4ede-c1f9-4a2b-8130-17144013bf31-kube-api-access-z2g77\" (UniqueName: \"kubernetes.io/projected/396c4ede-c1f9-4a2b-8130-17144013bf31-kube-api-access-z2g77\") pod \"396c4ede-c1f9-4a2b-8130-17144013bf31\" (UID: \"396c4ede-c1f9-4a2b-8130-17144013bf31\") " Nov 03 20:32:24.130110 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.129993 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-openshift-ca\") pod \"396c4ede-c1f9-4a2b-8130-17144013bf31\" (UID: \"396c4ede-c1f9-4a2b-8130-17144013bf31\") " Nov 03 20:32:24.130110 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.130046 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-xks-ca\") pod \"396c4ede-c1f9-4a2b-8130-17144013bf31\" (UID: \"396c4ede-c1f9-4a2b-8130-17144013bf31\") " Nov 03 20:32:24.130478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.130449 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b922a6-skwnm\" (UID: \"33e11096-489d-4ab1-9340-785ce05810ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" Nov 03 20:32:24.130744 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.130724 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b922a6-skwnm\" (UID: \"33e11096-489d-4ab1-9340-785ce05810ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" Nov 03 20:32:24.130975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.130495 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-openshift-ca" pod "396c4ede-c1f9-4a2b-8130-17144013bf31" (UID: "396c4ede-c1f9-4a2b-8130-17144013bf31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:24.131093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.131058 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9dd7\" (UniqueName: \"kubernetes.io/projected/33e11096-489d-4ab1-9340-785ce05810ee-kube-api-access-g9dd7\") pod \"maas-api-revoke-sub-14b922a6-skwnm\" (UID: \"33e11096-489d-4ab1-9340-785ce05810ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" Nov 03 20:32:24.131635 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.131612 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:24.143253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.143170 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-xks-ca" pod "396c4ede-c1f9-4a2b-8130-17144013bf31" (UID: "396c4ede-c1f9-4a2b-8130-17144013bf31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:24.186409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.160478 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396c4ede-c1f9-4a2b-8130-17144013bf31-kube-api-access-z2g77" pod "396c4ede-c1f9-4a2b-8130-17144013bf31" (UID: "396c4ede-c1f9-4a2b-8130-17144013bf31"). InnerVolumeSpecName "kube-api-access-z2g77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:24.234576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.234532 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b922a6-skwnm\" (UID: \"33e11096-489d-4ab1-9340-785ce05810ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" Nov 03 20:32:24.234740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.234678 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9dd7\" (UniqueName: \"kubernetes.io/projected/33e11096-489d-4ab1-9340-785ce05810ee-kube-api-access-g9dd7\") pod \"maas-api-revoke-sub-14b922a6-skwnm\" (UID: \"33e11096-489d-4ab1-9340-785ce05810ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" Nov 03 20:32:24.234955 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.234940 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b922a6-skwnm\" (UID: \"33e11096-489d-4ab1-9340-785ce05810ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" Nov 03 20:32:24.235097 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.235081 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2g77\" (UniqueName: \"kubernetes.io/projected/396c4ede-c1f9-4a2b-8130-17144013bf31-kube-api-access-z2g77\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:24.235142 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.235105 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396c4ede-c1f9-4a2b-8130-17144013bf31-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:24.235315 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.235278 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b922a6-skwnm\" (UID: \"33e11096-489d-4ab1-9340-785ce05810ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" Nov 03 20:32:24.237781 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.237758 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b922a6-skwnm\" (UID: \"33e11096-489d-4ab1-9340-785ce05810ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" Nov 03 20:32:24.247318 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.247291 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9dd7\" (UniqueName: \"kubernetes.io/projected/33e11096-489d-4ab1-9340-785ce05810ee-kube-api-access-g9dd7\") pod \"maas-api-revoke-sub-14b922a6-skwnm\" (UID: \"33e11096-489d-4ab1-9340-785ce05810ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" Nov 03 20:32:24.352199 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.352164 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" Nov 03 20:32:24.647101 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:32:24.647069 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e11096_489d_4ab1_9340_785ce05810ee.slice/crio-438bbb5c614dfa0c522172e5f4222506f673eaec4118de6e3633ba51a2defe09 WatchSource:0}: Error finding container 438bbb5c614dfa0c522172e5f4222506f673eaec4118de6e3633ba51a2defe09: Status 404 returned error can't find the container with id 438bbb5c614dfa0c522172e5f4222506f673eaec4118de6e3633ba51a2defe09 Nov 03 20:32:24.647971 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.647941 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm"] Nov 03 20:32:24.834694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.834666 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" Nov 03 20:32:24.835133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.834695 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w" event={"ID":"396c4ede-c1f9-4a2b-8130-17144013bf31","Type":"ContainerDied","Data":"d708b79c69a48e78d78ed97b9c4416449b78335b12c314208e5785ef4e040a56"} Nov 03 20:32:24.835133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.834746 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d708b79c69a48e78d78ed97b9c4416449b78335b12c314208e5785ef4e040a56" Nov 03 20:32:24.837439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:24.837410 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" event={"ID":"33e11096-489d-4ab1-9340-785ce05810ee","Type":"ContainerStarted","Data":"438bbb5c614dfa0c522172e5f4222506f673eaec4118de6e3633ba51a2defe09"} Nov 03 20:32:25.769158 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:25.769108 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32389e55-4f71-49b0-ba54-d53d241843b1" path="/var/lib/kubelet/pods/32389e55-4f71-49b0-ba54-d53d241843b1/volumes" Nov 03 20:32:25.849832 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:25.849793 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" event={"ID":"33e11096-489d-4ab1-9340-785ce05810ee","Type":"ContainerStarted","Data":"81cc399d6bbe330f89151e88555c08d70bdf9a18f52f592c4b657aced8951e5e"} Nov 03 20:32:25.877733 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:25.877666 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" podStartSLOduration=2.877643357 podStartE2EDuration="2.877643357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:25.874277684 +0000 UTC m=+2124.249874432" watchObservedRunningTime="2026-11-03 20:32:25.877643357 +0000 UTC m=+2124.253240105" Nov 03 20:32:26.862366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:26.862327 4457 generic.go:356] "Generic (PLEG): container finished" podID="33e11096-489d-4ab1-9340-785ce05810ee" containerID="81cc399d6bbe330f89151e88555c08d70bdf9a18f52f592c4b657aced8951e5e" exitCode=0 Nov 03 20:32:26.862871 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:26.862446 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" event={"ID":"33e11096-489d-4ab1-9340-785ce05810ee","Type":"ContainerDied","Data":"81cc399d6bbe330f89151e88555c08d70bdf9a18f52f592c4b657aced8951e5e"} Nov 03 20:32:28.073591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.073557 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" Nov 03 20:32:28.207491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.207366 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-xks-ca\") pod \"33e11096-489d-4ab1-9340-785ce05810ee\" (UID: \"33e11096-489d-4ab1-9340-785ce05810ee\") " Nov 03 20:32:28.207692 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.207672 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-openshift-ca\") pod \"33e11096-489d-4ab1-9340-785ce05810ee\" (UID: \"33e11096-489d-4ab1-9340-785ce05810ee\") " Nov 03 20:32:28.207787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.207772 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e11096-489d-4ab1-9340-785ce05810ee-kube-api-access-g9dd7\" (UniqueName: \"kubernetes.io/projected/33e11096-489d-4ab1-9340-785ce05810ee-kube-api-access-g9dd7\") pod \"33e11096-489d-4ab1-9340-785ce05810ee\" (UID: \"33e11096-489d-4ab1-9340-785ce05810ee\") " Nov 03 20:32:28.208024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.208000 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-openshift-ca" pod "33e11096-489d-4ab1-9340-785ce05810ee" (UID: "33e11096-489d-4ab1-9340-785ce05810ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:28.208427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.208400 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:28.209554 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.209527 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-xks-ca" pod "33e11096-489d-4ab1-9340-785ce05810ee" (UID: "33e11096-489d-4ab1-9340-785ce05810ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:28.209900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.209860 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e11096-489d-4ab1-9340-785ce05810ee-kube-api-access-g9dd7" pod "33e11096-489d-4ab1-9340-785ce05810ee" (UID: "33e11096-489d-4ab1-9340-785ce05810ee"). InnerVolumeSpecName "kube-api-access-g9dd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:28.309915 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.309880 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9dd7\" (UniqueName: \"kubernetes.io/projected/33e11096-489d-4ab1-9340-785ce05810ee-kube-api-access-g9dd7\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:28.309915 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.309913 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33e11096-489d-4ab1-9340-785ce05810ee-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:28.716686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.716645 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv"] Nov 03 20:32:28.749779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.749741 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j"] Nov 03 20:32:28.891542 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.891493 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" Nov 03 20:32:28.891780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.891601 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm" event={"ID":"33e11096-489d-4ab1-9340-785ce05810ee","Type":"ContainerDied","Data":"438bbb5c614dfa0c522172e5f4222506f673eaec4118de6e3633ba51a2defe09"} Nov 03 20:32:28.892146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:28.892124 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="438bbb5c614dfa0c522172e5f4222506f673eaec4118de6e3633ba51a2defe09" Nov 03 20:32:31.121522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.120449 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7"] Nov 03 20:32:31.122155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.121978 4457 cpu_manager.go:414] "containerMap: removing container" podUID="33e11096-489d-4ab1-9340-785ce05810ee" containerName="revoke-keys" Nov 03 20:32:31.122155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.122002 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e11096-489d-4ab1-9340-785ce05810ee" containerName="revoke-keys" Nov 03 20:32:31.122155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.122053 4457 cpu_manager.go:414] "containerMap: removing container" podUID="396c4ede-c1f9-4a2b-8130-17144013bf31" containerName="revoke-keys" Nov 03 20:32:31.122155 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.122061 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396c4ede-c1f9-4a2b-8130-17144013bf31" containerName="revoke-keys" Nov 03 20:32:31.122431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.122232 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e11096-489d-4ab1-9340-785ce05810ee" containerName="revoke-keys" Nov 03 20:32:31.122431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.122256 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="396c4ede-c1f9-4a2b-8130-17144013bf31" containerName="revoke-keys" Nov 03 20:32:31.126739 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.126709 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" Nov 03 20:32:31.130299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.130259 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7"] Nov 03 20:32:31.133305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.133284 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:32:31.133552 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.133534 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:32:31.291275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.291228 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-206f0544-nbbm7\" (UID: \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" Nov 03 20:32:31.291515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.291429 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-206f0544-nbbm7\" (UID: \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" Nov 03 20:32:31.291676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.291656 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9hm\" (UniqueName: \"kubernetes.io/projected/ba338dc2-d4f7-45b5-ad98-a8a76121203d-kube-api-access-nw9hm\") pod \"maas-api-revoke-sub-206f0544-nbbm7\" (UID: \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" Nov 03 20:32:31.393561 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.393453 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-206f0544-nbbm7\" (UID: \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" Nov 03 20:32:31.393719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.393676 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9hm\" (UniqueName: \"kubernetes.io/projected/ba338dc2-d4f7-45b5-ad98-a8a76121203d-kube-api-access-nw9hm\") pod \"maas-api-revoke-sub-206f0544-nbbm7\" (UID: \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" Nov 03 20:32:31.393814 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.393799 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-206f0544-nbbm7\" (UID: \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" Nov 03 20:32:31.394697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.394665 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-206f0544-nbbm7\" (UID: \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" Nov 03 20:32:31.396343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.396318 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-206f0544-nbbm7\" (UID: \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" Nov 03 20:32:31.407837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.407811 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9hm\" (UniqueName: \"kubernetes.io/projected/ba338dc2-d4f7-45b5-ad98-a8a76121203d-kube-api-access-nw9hm\") pod \"maas-api-revoke-sub-206f0544-nbbm7\" (UID: \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" Nov 03 20:32:31.486313 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.486269 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" Nov 03 20:32:31.793047 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:32:31.793011 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba338dc2_d4f7_45b5_ad98_a8a76121203d.slice/crio-2dc39aee399699412954786419ef6675794d1ed60528d217dc7dbfdc1fde7fba WatchSource:0}: Error finding container 2dc39aee399699412954786419ef6675794d1ed60528d217dc7dbfdc1fde7fba: Status 404 returned error can't find the container with id 2dc39aee399699412954786419ef6675794d1ed60528d217dc7dbfdc1fde7fba Nov 03 20:32:31.793536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.793510 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7"] Nov 03 20:32:31.947413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:31.932437 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" event={"ID":"ba338dc2-d4f7-45b5-ad98-a8a76121203d","Type":"ContainerStarted","Data":"2dc39aee399699412954786419ef6675794d1ed60528d217dc7dbfdc1fde7fba"} Nov 03 20:32:32.944282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:32.944190 4457 generic.go:356] "Generic (PLEG): container finished" podID="ba338dc2-d4f7-45b5-ad98-a8a76121203d" containerID="26d2927a1652c541e80ec59fa6854c1e336b447531bc614f86d7d771d8d33f21" exitCode=0 Nov 03 20:32:32.944778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:32.944276 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" event={"ID":"ba338dc2-d4f7-45b5-ad98-a8a76121203d","Type":"ContainerDied","Data":"26d2927a1652c541e80ec59fa6854c1e336b447531bc614f86d7d771d8d33f21"} Nov 03 20:32:33.718420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:33.717932 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" podUID="2243a6fb-07fb-4f03-ba5f-76d5909e7a44" containerName="payload-processing" containerID="cri-o://54e904e1ab51e4388649c5f4e37c16d47dcaaba31be19c4eb81d66e09d874b5b" gracePeriod=25 Nov 03 20:32:33.758007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:33.757946 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" podUID="9899751b-58b7-428e-a0b1-464a005dfabe" containerName="payload-pre-processing" containerID="cri-o://863ec92fefd1eb2585e90128af0770cfe26667573146e522acb5356a215cdf7d" gracePeriod=25 Nov 03 20:32:33.956956 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:33.956919 4457 generic.go:356] "Generic (PLEG): container finished" podID="2243a6fb-07fb-4f03-ba5f-76d5909e7a44" containerID="54e904e1ab51e4388649c5f4e37c16d47dcaaba31be19c4eb81d66e09d874b5b" exitCode=0 Nov 03 20:32:33.957460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:33.957240 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" event={"ID":"2243a6fb-07fb-4f03-ba5f-76d5909e7a44","Type":"ContainerDied","Data":"54e904e1ab51e4388649c5f4e37c16d47dcaaba31be19c4eb81d66e09d874b5b"} Nov 03 20:32:34.057077 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.057048 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" Nov 03 20:32:34.154312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.154272 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-kube-api-access-qqhb9\" (UniqueName: \"kubernetes.io/projected/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-kube-api-access-qqhb9\") pod \"2243a6fb-07fb-4f03-ba5f-76d5909e7a44\" (UID: \"2243a6fb-07fb-4f03-ba5f-76d5909e7a44\") " Nov 03 20:32:34.154528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.154477 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-plugins-config-volume\") pod \"2243a6fb-07fb-4f03-ba5f-76d5909e7a44\" (UID: \"2243a6fb-07fb-4f03-ba5f-76d5909e7a44\") " Nov 03 20:32:34.155019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.154978 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-plugins-config-volume" pod "2243a6fb-07fb-4f03-ba5f-76d5909e7a44" (UID: "2243a6fb-07fb-4f03-ba5f-76d5909e7a44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:34.155239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.155216 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:34.156910 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.156880 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-kube-api-access-qqhb9" pod "2243a6fb-07fb-4f03-ba5f-76d5909e7a44" (UID: "2243a6fb-07fb-4f03-ba5f-76d5909e7a44"). InnerVolumeSpecName "kube-api-access-qqhb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:34.201851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.201823 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" Nov 03 20:32:34.257280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.257246 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-xks-ca\") pod \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\" (UID: \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\") " Nov 03 20:32:34.257494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.257463 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba338dc2-d4f7-45b5-ad98-a8a76121203d-kube-api-access-nw9hm\" (UniqueName: \"kubernetes.io/projected/ba338dc2-d4f7-45b5-ad98-a8a76121203d-kube-api-access-nw9hm\") pod \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\" (UID: \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\") " Nov 03 20:32:34.257877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.257845 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-openshift-ca\") pod \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\" (UID: \"ba338dc2-d4f7-45b5-ad98-a8a76121203d\") " Nov 03 20:32:34.258524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.258495 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-openshift-ca" pod "ba338dc2-d4f7-45b5-ad98-a8a76121203d" (UID: "ba338dc2-d4f7-45b5-ad98-a8a76121203d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:34.259326 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.259301 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:34.259434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.259331 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqhb9\" (UniqueName: \"kubernetes.io/projected/2243a6fb-07fb-4f03-ba5f-76d5909e7a44-kube-api-access-qqhb9\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:34.259827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.259806 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-xks-ca" pod "ba338dc2-d4f7-45b5-ad98-a8a76121203d" (UID: "ba338dc2-d4f7-45b5-ad98-a8a76121203d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:34.260043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.260028 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba338dc2-d4f7-45b5-ad98-a8a76121203d-kube-api-access-nw9hm" pod "ba338dc2-d4f7-45b5-ad98-a8a76121203d" (UID: "ba338dc2-d4f7-45b5-ad98-a8a76121203d"). InnerVolumeSpecName "kube-api-access-nw9hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:34.363411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.361834 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba338dc2-d4f7-45b5-ad98-a8a76121203d-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:34.363411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.361888 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw9hm\" (UniqueName: \"kubernetes.io/projected/ba338dc2-d4f7-45b5-ad98-a8a76121203d-kube-api-access-nw9hm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:34.970156 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.969755 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" Nov 03 20:32:34.970156 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.969762 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7" event={"ID":"ba338dc2-d4f7-45b5-ad98-a8a76121203d","Type":"ContainerDied","Data":"2dc39aee399699412954786419ef6675794d1ed60528d217dc7dbfdc1fde7fba"} Nov 03 20:32:34.970156 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.969922 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dc39aee399699412954786419ef6675794d1ed60528d217dc7dbfdc1fde7fba" Nov 03 20:32:34.972574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.972541 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" event={"ID":"2243a6fb-07fb-4f03-ba5f-76d5909e7a44","Type":"ContainerDied","Data":"c1febcefa3dca3d77c86dbc976d3859f2fca045df553517650950da6d46c0513"} Nov 03 20:32:34.972753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.972575 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv" Nov 03 20:32:34.972753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:34.972598 4457 scope.go:122] "RemoveContainer" containerID="54e904e1ab51e4388649c5f4e37c16d47dcaaba31be19c4eb81d66e09d874b5b" Nov 03 20:32:35.006753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:35.006684 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv"] Nov 03 20:32:35.012674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:35.012638 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-34e844-599b659bb-b52jv"] Nov 03 20:32:35.763198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:35.763160 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2243a6fb-07fb-4f03-ba5f-76d5909e7a44" path="/var/lib/kubelet/pods/2243a6fb-07fb-4f03-ba5f-76d5909e7a44/volumes" Nov 03 20:32:36.150116 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:32:36.150079 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:32:36.525431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:36.525294 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" podUID="9899751b-58b7-428e-a0b1-464a005dfabe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:32:38.382530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.382475 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf"] Nov 03 20:32:38.383723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.383693 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2243a6fb-07fb-4f03-ba5f-76d5909e7a44" containerName="payload-processing" Nov 03 20:32:38.383723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.383722 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2243a6fb-07fb-4f03-ba5f-76d5909e7a44" containerName="payload-processing" Nov 03 20:32:38.383951 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.383775 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ba338dc2-d4f7-45b5-ad98-a8a76121203d" containerName="revoke-keys" Nov 03 20:32:38.383951 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.383784 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba338dc2-d4f7-45b5-ad98-a8a76121203d" containerName="revoke-keys" Nov 03 20:32:38.384061 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.384017 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="2243a6fb-07fb-4f03-ba5f-76d5909e7a44" containerName="payload-processing" Nov 03 20:32:38.384061 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.384042 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba338dc2-d4f7-45b5-ad98-a8a76121203d" containerName="revoke-keys" Nov 03 20:32:38.400407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.393053 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" Nov 03 20:32:38.400407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.398470 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:32:38.400407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.398509 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:32:38.401695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.401300 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf"] Nov 03 20:32:38.487601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.485251 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" podUID="a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" containerName="main" containerID="cri-o://3e1d405ac8674758d132a1f74b4d5f9bfe309adce1f8227f82339b759127f348" gracePeriod=45 Nov 03 20:32:38.569924 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.569888 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nxmv\" (UniqueName: \"kubernetes.io/projected/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-kube-api-access-9nxmv\") pod \"maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf\" (UID: \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" Nov 03 20:32:38.570103 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.569946 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf\" (UID: \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" Nov 03 20:32:38.570165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.570135 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf\" (UID: \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" Nov 03 20:32:38.680462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.678846 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nxmv\" (UniqueName: \"kubernetes.io/projected/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-kube-api-access-9nxmv\") pod \"maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf\" (UID: \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" Nov 03 20:32:38.680462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.678919 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf\" (UID: \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" Nov 03 20:32:38.680462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.679076 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf\" (UID: \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" Nov 03 20:32:38.680462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.680199 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf\" (UID: \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" Nov 03 20:32:38.703125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.698068 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf\" (UID: \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" Nov 03 20:32:38.709514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.704381 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nxmv\" (UniqueName: \"kubernetes.io/projected/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-kube-api-access-9nxmv\") pod \"maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf\" (UID: \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" Nov 03 20:32:38.786453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.786418 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" Nov 03 20:32:38.846124 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.846094 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:32:38.995266 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.995181 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-model-cache\") pod \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " Nov 03 20:32:38.995266 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.995242 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-kube-api-access-25tbn\" (UniqueName: \"kubernetes.io/projected/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-kube-api-access-25tbn\") pod \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " Nov 03 20:32:38.995566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.995544 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-home\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-home\") pod \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " Nov 03 20:32:38.995647 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.995604 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tmp-dir\") pod \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " Nov 03 20:32:38.995706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.995646 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-dshm\") pod \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " Nov 03 20:32:38.995827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.995809 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tls-certs\") pod \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\" (UID: \"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e\") " Nov 03 20:32:38.996322 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.995806 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-model-cache" pod "a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" (UID: "a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:38.996412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.995830 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-home" pod "a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" (UID: "a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:38.996412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.996002 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tmp-dir" pod "a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" (UID: "a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:38.996714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.996692 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:38.996801 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.996719 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:38.996801 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.996733 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:38.998735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.998706 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tls-certs" pod "a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" (UID: "a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:38.998735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.998724 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-dshm" pod "a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" (UID: "a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:38.998883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:38.998765 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-kube-api-access-25tbn" pod "a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" (UID: "a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e"). InnerVolumeSpecName "kube-api-access-25tbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:39.029319 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.029282 4457 generic.go:356] "Generic (PLEG): container finished" podID="a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" containerID="3e1d405ac8674758d132a1f74b4d5f9bfe309adce1f8227f82339b759127f348" exitCode=0 Nov 03 20:32:39.029524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.029333 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" Nov 03 20:32:39.029524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.029355 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" event={"ID":"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e","Type":"ContainerDied","Data":"3e1d405ac8674758d132a1f74b4d5f9bfe309adce1f8227f82339b759127f348"} Nov 03 20:32:39.029524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.029444 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" event={"ID":"a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e","Type":"ContainerDied","Data":"5d639ba72bcc7c517389e5d2b9e3e5a29863d56298059385d8c267a5960794ae"} Nov 03 20:32:39.029524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.029469 4457 scope.go:122] "RemoveContainer" containerID="3e1d405ac8674758d132a1f74b4d5f9bfe309adce1f8227f82339b759127f348" Nov 03 20:32:39.042541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.042493 4457 scope.go:122] "RemoveContainer" containerID="3e1d405ac8674758d132a1f74b4d5f9bfe309adce1f8227f82339b759127f348" Nov 03 20:32:39.042868 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:32:39.042839 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1d405ac8674758d132a1f74b4d5f9bfe309adce1f8227f82339b759127f348\": container with ID starting with 3e1d405ac8674758d132a1f74b4d5f9bfe309adce1f8227f82339b759127f348 not found: ID does not exist" containerID="3e1d405ac8674758d132a1f74b4d5f9bfe309adce1f8227f82339b759127f348" Nov 03 20:32:39.042979 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.042873 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1d405ac8674758d132a1f74b4d5f9bfe309adce1f8227f82339b759127f348"} err="failed to get container status \"3e1d405ac8674758d132a1f74b4d5f9bfe309adce1f8227f82339b759127f348\": rpc error: code = NotFound desc = could not find container \"3e1d405ac8674758d132a1f74b4d5f9bfe309adce1f8227f82339b759127f348\": container with ID starting with 3e1d405ac8674758d132a1f74b4d5f9bfe309adce1f8227f82339b759127f348 not found: ID does not exist" Nov 03 20:32:39.058273 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.058248 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4"] Nov 03 20:32:39.063548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.063518 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4"] Nov 03 20:32:39.083162 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:32:39.083128 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78469a88_9d85_4e50_bec8_efdb5fdc1a5e.slice/crio-9afdd7ce3ebdedb2419a93104999d8f13af6ce089d0607a700c86536ada4f824 WatchSource:0}: Error finding container 9afdd7ce3ebdedb2419a93104999d8f13af6ce089d0607a700c86536ada4f824: Status 404 returned error can't find the container with id 9afdd7ce3ebdedb2419a93104999d8f13af6ce089d0607a700c86536ada4f824 Nov 03 20:32:39.084703 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.084675 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf"] Nov 03 20:32:39.098560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.098537 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:39.098560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.098562 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25tbn\" (UniqueName: \"kubernetes.io/projected/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-kube-api-access-25tbn\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:39.098726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.098573 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:39.765861 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.765828 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" path="/var/lib/kubelet/pods/a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e/volumes" Nov 03 20:32:39.800037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:39.799997 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7ae0f1/test-model-7ae0f1-kserve-84db68d48c-f8dr4" podUID="a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.193:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:32:40.042714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:40.042610 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" event={"ID":"78469a88-9d85-4e50-bec8-efdb5fdc1a5e","Type":"ContainerStarted","Data":"8d51264cdf77f69c73cfc1c9bb608d61ac87b20191893a1d9c6455057665df13"} Nov 03 20:32:40.042714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:40.042668 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" event={"ID":"78469a88-9d85-4e50-bec8-efdb5fdc1a5e","Type":"ContainerStarted","Data":"9afdd7ce3ebdedb2419a93104999d8f13af6ce089d0607a700c86536ada4f824"} Nov 03 20:32:40.069207 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:40.069135 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" podStartSLOduration=2.069115894 podStartE2EDuration="2.069115894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:40.06106953 +0000 UTC m=+2138.436666280" watchObservedRunningTime="2026-11-03 20:32:40.069115894 +0000 UTC m=+2138.444712642" Nov 03 20:32:41.069499 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:41.069462 4457 generic.go:356] "Generic (PLEG): container finished" podID="78469a88-9d85-4e50-bec8-efdb5fdc1a5e" containerID="8d51264cdf77f69c73cfc1c9bb608d61ac87b20191893a1d9c6455057665df13" exitCode=0 Nov 03 20:32:41.070069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:41.069539 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" event={"ID":"78469a88-9d85-4e50-bec8-efdb5fdc1a5e","Type":"ContainerDied","Data":"8d51264cdf77f69c73cfc1c9bb608d61ac87b20191893a1d9c6455057665df13"} Nov 03 20:32:42.160441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.160332 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq"] Nov 03 20:32:42.161632 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.161602 4457 cpu_manager.go:414] "containerMap: removing container" podUID="a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" containerName="main" Nov 03 20:32:42.161632 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.161631 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" containerName="main" Nov 03 20:32:42.161917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.161901 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d8c64e-4b95-4b73-bd9d-4183bf9a9c1e" containerName="main" Nov 03 20:32:42.215442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.214104 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq"] Nov 03 20:32:42.215442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.214247 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" Nov 03 20:32:42.276081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.275538 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17905aef-859b-48d6-8bdc-46525adea158-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq\" (UID: \"17905aef-859b-48d6-8bdc-46525adea158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" Nov 03 20:32:42.276081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.275847 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww4g6\" (UniqueName: \"kubernetes.io/projected/17905aef-859b-48d6-8bdc-46525adea158-kube-api-access-ww4g6\") pod \"maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq\" (UID: \"17905aef-859b-48d6-8bdc-46525adea158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" Nov 03 20:32:42.276081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.275900 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17905aef-859b-48d6-8bdc-46525adea158-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq\" (UID: \"17905aef-859b-48d6-8bdc-46525adea158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" Nov 03 20:32:42.377399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.377337 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17905aef-859b-48d6-8bdc-46525adea158-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq\" (UID: \"17905aef-859b-48d6-8bdc-46525adea158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" Nov 03 20:32:42.377704 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.377681 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4g6\" (UniqueName: \"kubernetes.io/projected/17905aef-859b-48d6-8bdc-46525adea158-kube-api-access-ww4g6\") pod \"maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq\" (UID: \"17905aef-859b-48d6-8bdc-46525adea158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" Nov 03 20:32:42.377815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.377741 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17905aef-859b-48d6-8bdc-46525adea158-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq\" (UID: \"17905aef-859b-48d6-8bdc-46525adea158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" Nov 03 20:32:42.378113 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.378092 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17905aef-859b-48d6-8bdc-46525adea158-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq\" (UID: \"17905aef-859b-48d6-8bdc-46525adea158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" Nov 03 20:32:42.381918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.381894 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17905aef-859b-48d6-8bdc-46525adea158-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq\" (UID: \"17905aef-859b-48d6-8bdc-46525adea158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" Nov 03 20:32:42.390944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.390871 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww4g6\" (UniqueName: \"kubernetes.io/projected/17905aef-859b-48d6-8bdc-46525adea158-kube-api-access-ww4g6\") pod \"maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq\" (UID: \"17905aef-859b-48d6-8bdc-46525adea158\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" Nov 03 20:32:42.431528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.431499 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" Nov 03 20:32:42.482443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.481359 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-kube-api-access-9nxmv\" (UniqueName: \"kubernetes.io/projected/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-kube-api-access-9nxmv\") pod \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\" (UID: \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\") " Nov 03 20:32:42.482443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.481511 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-openshift-ca\") pod \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\" (UID: \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\") " Nov 03 20:32:42.482443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.481722 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-xks-ca\") pod \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\" (UID: \"78469a88-9d85-4e50-bec8-efdb5fdc1a5e\") " Nov 03 20:32:42.483723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.483694 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-openshift-ca" pod "78469a88-9d85-4e50-bec8-efdb5fdc1a5e" (UID: "78469a88-9d85-4e50-bec8-efdb5fdc1a5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:42.507923 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.507882 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-kube-api-access-9nxmv" pod "78469a88-9d85-4e50-bec8-efdb5fdc1a5e" (UID: "78469a88-9d85-4e50-bec8-efdb5fdc1a5e"). InnerVolumeSpecName "kube-api-access-9nxmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:42.510186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.510148 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-xks-ca" pod "78469a88-9d85-4e50-bec8-efdb5fdc1a5e" (UID: "78469a88-9d85-4e50-bec8-efdb5fdc1a5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:42.535370 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.535339 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" Nov 03 20:32:42.585749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.583733 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nxmv\" (UniqueName: \"kubernetes.io/projected/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-kube-api-access-9nxmv\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:42.585749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.583770 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:42.585749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.583786 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78469a88-9d85-4e50-bec8-efdb5fdc1a5e-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:42.841322 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:32:42.841277 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17905aef_859b_48d6_8bdc_46525adea158.slice/crio-36a2571c04e4537eca6bd0843f40f58557f301932e39ba905b8aaed94c162c98 WatchSource:0}: Error finding container 36a2571c04e4537eca6bd0843f40f58557f301932e39ba905b8aaed94c162c98: Status 404 returned error can't find the container with id 36a2571c04e4537eca6bd0843f40f58557f301932e39ba905b8aaed94c162c98 Nov 03 20:32:42.843327 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:42.843302 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq"] Nov 03 20:32:43.236434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:43.226343 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" event={"ID":"78469a88-9d85-4e50-bec8-efdb5fdc1a5e","Type":"ContainerDied","Data":"9afdd7ce3ebdedb2419a93104999d8f13af6ce089d0607a700c86536ada4f824"} Nov 03 20:32:43.236434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:43.226426 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9afdd7ce3ebdedb2419a93104999d8f13af6ce089d0607a700c86536ada4f824" Nov 03 20:32:43.236434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:43.226510 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf" Nov 03 20:32:43.245452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:43.245351 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" event={"ID":"17905aef-859b-48d6-8bdc-46525adea158","Type":"ContainerStarted","Data":"36a2571c04e4537eca6bd0843f40f58557f301932e39ba905b8aaed94c162c98"} Nov 03 20:32:43.429655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:43.429620 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k"] Nov 03 20:32:43.435145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:43.435115 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-34e844-2s27k"] Nov 03 20:32:43.765294 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:43.765257 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38264330-3138-4328-95a7-de830575c54f" path="/var/lib/kubelet/pods/38264330-3138-4328-95a7-de830575c54f/volumes" Nov 03 20:32:44.260074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:44.260028 4457 generic.go:356] "Generic (PLEG): container finished" podID="17905aef-859b-48d6-8bdc-46525adea158" containerID="83858d8a9d4e22e6f1865b806db47d3957ce58f6638447dfa572d82e4e969834" exitCode=0 Nov 03 20:32:44.260538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:44.260182 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" event={"ID":"17905aef-859b-48d6-8bdc-46525adea158","Type":"ContainerDied","Data":"83858d8a9d4e22e6f1865b806db47d3957ce58f6638447dfa572d82e4e969834"} Nov 03 20:32:45.515010 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.514979 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56"] Nov 03 20:32:45.709508 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.708955 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" Nov 03 20:32:45.736424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.733428 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b"] Nov 03 20:32:45.736424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.734741 4457 cpu_manager.go:414] "containerMap: removing container" podUID="78469a88-9d85-4e50-bec8-efdb5fdc1a5e" containerName="revoke-keys" Nov 03 20:32:45.736424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.734764 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78469a88-9d85-4e50-bec8-efdb5fdc1a5e" containerName="revoke-keys" Nov 03 20:32:45.736424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.734787 4457 cpu_manager.go:414] "containerMap: removing container" podUID="17905aef-859b-48d6-8bdc-46525adea158" containerName="revoke-keys" Nov 03 20:32:45.736424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.734796 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17905aef-859b-48d6-8bdc-46525adea158" containerName="revoke-keys" Nov 03 20:32:45.736424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.735002 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="78469a88-9d85-4e50-bec8-efdb5fdc1a5e" containerName="revoke-keys" Nov 03 20:32:45.736424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.735016 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="17905aef-859b-48d6-8bdc-46525adea158" containerName="revoke-keys" Nov 03 20:32:45.761745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.759062 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" Nov 03 20:32:45.788525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.787261 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b"] Nov 03 20:32:45.797711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.797533 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17905aef-859b-48d6-8bdc-46525adea158-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17905aef-859b-48d6-8bdc-46525adea158-maas-api-xks-ca\") pod \"17905aef-859b-48d6-8bdc-46525adea158\" (UID: \"17905aef-859b-48d6-8bdc-46525adea158\") " Nov 03 20:32:45.797711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.797629 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17905aef-859b-48d6-8bdc-46525adea158-kube-api-access-ww4g6\" (UniqueName: \"kubernetes.io/projected/17905aef-859b-48d6-8bdc-46525adea158-kube-api-access-ww4g6\") pod \"17905aef-859b-48d6-8bdc-46525adea158\" (UID: \"17905aef-859b-48d6-8bdc-46525adea158\") " Nov 03 20:32:45.797893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.797738 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17905aef-859b-48d6-8bdc-46525adea158-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17905aef-859b-48d6-8bdc-46525adea158-maas-api-openshift-ca\") pod \"17905aef-859b-48d6-8bdc-46525adea158\" (UID: \"17905aef-859b-48d6-8bdc-46525adea158\") " Nov 03 20:32:45.809009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.798879 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17905aef-859b-48d6-8bdc-46525adea158-maas-api-openshift-ca" pod "17905aef-859b-48d6-8bdc-46525adea158" (UID: "17905aef-859b-48d6-8bdc-46525adea158"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:45.809009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.804973 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17905aef-859b-48d6-8bdc-46525adea158-kube-api-access-ww4g6" pod "17905aef-859b-48d6-8bdc-46525adea158" (UID: "17905aef-859b-48d6-8bdc-46525adea158"). InnerVolumeSpecName "kube-api-access-ww4g6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:45.809009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.808600 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17905aef-859b-48d6-8bdc-46525adea158-maas-api-xks-ca" pod "17905aef-859b-48d6-8bdc-46525adea158" (UID: "17905aef-859b-48d6-8bdc-46525adea158"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:45.900496 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.900446 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chn5r\" (UniqueName: \"kubernetes.io/projected/a618b730-ae49-42ce-ae44-aabfdd1a19cf-kube-api-access-chn5r\") pod \"maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b\" (UID: \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" Nov 03 20:32:45.900707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.900582 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b\" (UID: \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" Nov 03 20:32:45.900707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.900637 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b\" (UID: \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" Nov 03 20:32:45.902478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.902348 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17905aef-859b-48d6-8bdc-46525adea158-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:45.905511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.902631 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww4g6\" (UniqueName: \"kubernetes.io/projected/17905aef-859b-48d6-8bdc-46525adea158-kube-api-access-ww4g6\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:45.905511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:45.902664 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17905aef-859b-48d6-8bdc-46525adea158-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:46.003883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:46.003844 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chn5r\" (UniqueName: \"kubernetes.io/projected/a618b730-ae49-42ce-ae44-aabfdd1a19cf-kube-api-access-chn5r\") pod \"maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b\" (UID: \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" Nov 03 20:32:46.004128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:46.003966 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b\" (UID: \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" Nov 03 20:32:46.004128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:46.004016 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b\" (UID: \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" Nov 03 20:32:46.004964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:46.004934 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b\" (UID: \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" Nov 03 20:32:46.007060 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:46.007032 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b\" (UID: \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" Nov 03 20:32:46.022408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:46.019227 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chn5r\" (UniqueName: \"kubernetes.io/projected/a618b730-ae49-42ce-ae44-aabfdd1a19cf-kube-api-access-chn5r\") pod \"maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b\" (UID: \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" Nov 03 20:32:46.130631 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:46.130585 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" Nov 03 20:32:46.294453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:46.293106 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" event={"ID":"17905aef-859b-48d6-8bdc-46525adea158","Type":"ContainerDied","Data":"36a2571c04e4537eca6bd0843f40f58557f301932e39ba905b8aaed94c162c98"} Nov 03 20:32:46.294453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:46.293170 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36a2571c04e4537eca6bd0843f40f58557f301932e39ba905b8aaed94c162c98" Nov 03 20:32:46.294453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:46.293240 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq" Nov 03 20:32:46.432971 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:32:46.432943 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda618b730_ae49_42ce_ae44_aabfdd1a19cf.slice/crio-315f36314b9eddb5be3f6022f81ac808b4f371ea87abd57c05a989c39932325e WatchSource:0}: Error finding container 315f36314b9eddb5be3f6022f81ac808b4f371ea87abd57c05a989c39932325e: Status 404 returned error can't find the container with id 315f36314b9eddb5be3f6022f81ac808b4f371ea87abd57c05a989c39932325e Nov 03 20:32:46.433521 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:46.433501 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b"] Nov 03 20:32:46.525310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:46.525270 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" podUID="9899751b-58b7-428e-a0b1-464a005dfabe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:32:47.307040 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:47.307002 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" event={"ID":"a618b730-ae49-42ce-ae44-aabfdd1a19cf","Type":"ContainerStarted","Data":"01b6be8012b0ce94d4e436b0152c6fbe9cc7e2247b2d6114af09a30363415efa"} Nov 03 20:32:47.307232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:47.307072 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" event={"ID":"a618b730-ae49-42ce-ae44-aabfdd1a19cf","Type":"ContainerStarted","Data":"315f36314b9eddb5be3f6022f81ac808b4f371ea87abd57c05a989c39932325e"} Nov 03 20:32:47.331488 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:47.331426 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" podStartSLOduration=2.331408454 podStartE2EDuration="2.331408454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:47.325212139 +0000 UTC m=+2145.700808886" watchObservedRunningTime="2026-11-03 20:32:47.331408454 +0000 UTC m=+2145.707005202" Nov 03 20:32:48.319198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:48.319161 4457 generic.go:356] "Generic (PLEG): container finished" podID="a618b730-ae49-42ce-ae44-aabfdd1a19cf" containerID="01b6be8012b0ce94d4e436b0152c6fbe9cc7e2247b2d6114af09a30363415efa" exitCode=0 Nov 03 20:32:48.319666 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:48.319243 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" event={"ID":"a618b730-ae49-42ce-ae44-aabfdd1a19cf","Type":"ContainerDied","Data":"01b6be8012b0ce94d4e436b0152c6fbe9cc7e2247b2d6114af09a30363415efa"} Nov 03 20:32:48.530822 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:48.530768 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz"] Nov 03 20:32:48.539669 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:48.539628 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff"] Nov 03 20:32:49.560912 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:49.560883 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" Nov 03 20:32:49.720614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:49.720517 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-xks-ca\") pod \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\" (UID: \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\") " Nov 03 20:32:49.720846 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:49.720828 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a618b730-ae49-42ce-ae44-aabfdd1a19cf-kube-api-access-chn5r\" (UniqueName: \"kubernetes.io/projected/a618b730-ae49-42ce-ae44-aabfdd1a19cf-kube-api-access-chn5r\") pod \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\" (UID: \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\") " Nov 03 20:32:49.721023 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:49.721007 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-openshift-ca\") pod \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\" (UID: \"a618b730-ae49-42ce-ae44-aabfdd1a19cf\") " Nov 03 20:32:49.721346 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:49.721319 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-openshift-ca" pod "a618b730-ae49-42ce-ae44-aabfdd1a19cf" (UID: "a618b730-ae49-42ce-ae44-aabfdd1a19cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:49.721673 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:49.721653 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:49.722812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:49.722784 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-xks-ca" pod "a618b730-ae49-42ce-ae44-aabfdd1a19cf" (UID: "a618b730-ae49-42ce-ae44-aabfdd1a19cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:49.723028 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:49.723008 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a618b730-ae49-42ce-ae44-aabfdd1a19cf-kube-api-access-chn5r" pod "a618b730-ae49-42ce-ae44-aabfdd1a19cf" (UID: "a618b730-ae49-42ce-ae44-aabfdd1a19cf"). InnerVolumeSpecName "kube-api-access-chn5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:49.823726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:49.823689 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chn5r\" (UniqueName: \"kubernetes.io/projected/a618b730-ae49-42ce-ae44-aabfdd1a19cf-kube-api-access-chn5r\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:49.823726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:49.823721 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a618b730-ae49-42ce-ae44-aabfdd1a19cf-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:50.341316 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:50.341288 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" Nov 03 20:32:50.341533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:50.341321 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b" event={"ID":"a618b730-ae49-42ce-ae44-aabfdd1a19cf","Type":"ContainerDied","Data":"315f36314b9eddb5be3f6022f81ac808b4f371ea87abd57c05a989c39932325e"} Nov 03 20:32:50.341533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:50.341397 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="315f36314b9eddb5be3f6022f81ac808b4f371ea87abd57c05a989c39932325e" Nov 03 20:32:51.221895 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.221845 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk"] Nov 03 20:32:51.222422 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.222105 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" podUID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" containerName="istio-proxy" containerID="cri-o://4dd34ca74dc2ca470921cdaaf15561ce5bbaebac9ff00968b6b051ec23e115eb" gracePeriod=30 Nov 03 20:32:51.530474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.525373 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6"] Nov 03 20:32:51.530474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.526118 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" podUID="e8bcbefd-88d8-4eae-9cf5-fa29620724a2" containerName="istio-proxy" containerID="cri-o://65b023e3e9738b2de64fd05f238c8b0e5bdb2efca4f00efd644f6af140f51850" gracePeriod=30 Nov 03 20:32:51.904436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.904379 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd"] Nov 03 20:32:51.905647 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.905617 4457 cpu_manager.go:414] "containerMap: removing container" podUID="a618b730-ae49-42ce-ae44-aabfdd1a19cf" containerName="revoke-keys" Nov 03 20:32:51.905647 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.905648 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a618b730-ae49-42ce-ae44-aabfdd1a19cf" containerName="revoke-keys" Nov 03 20:32:51.905917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.905903 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="a618b730-ae49-42ce-ae44-aabfdd1a19cf" containerName="revoke-keys" Nov 03 20:32:51.911515 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.911276 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" Nov 03 20:32:51.915510 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.915485 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:32:51.916722 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.916699 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:32:51.918607 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.918579 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd"] Nov 03 20:32:51.973674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.973617 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b991442-a874-43f1-bf91-001d877344eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd\" (UID: \"6b991442-a874-43f1-bf91-001d877344eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" Nov 03 20:32:51.974038 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.974012 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b991442-a874-43f1-bf91-001d877344eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd\" (UID: \"6b991442-a874-43f1-bf91-001d877344eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" Nov 03 20:32:51.974138 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:51.974122 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sztw8\" (UniqueName: \"kubernetes.io/projected/6b991442-a874-43f1-bf91-001d877344eb-kube-api-access-sztw8\") pod \"maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd\" (UID: \"6b991442-a874-43f1-bf91-001d877344eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" Nov 03 20:32:52.076160 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:52.076094 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b991442-a874-43f1-bf91-001d877344eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd\" (UID: \"6b991442-a874-43f1-bf91-001d877344eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" Nov 03 20:32:52.076492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:52.076220 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sztw8\" (UniqueName: \"kubernetes.io/projected/6b991442-a874-43f1-bf91-001d877344eb-kube-api-access-sztw8\") pod \"maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd\" (UID: \"6b991442-a874-43f1-bf91-001d877344eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" Nov 03 20:32:52.080415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:52.077026 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b991442-a874-43f1-bf91-001d877344eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd\" (UID: \"6b991442-a874-43f1-bf91-001d877344eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" Nov 03 20:32:52.080415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:52.077954 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b991442-a874-43f1-bf91-001d877344eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd\" (UID: \"6b991442-a874-43f1-bf91-001d877344eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" Nov 03 20:32:52.080415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:52.079741 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b991442-a874-43f1-bf91-001d877344eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd\" (UID: \"6b991442-a874-43f1-bf91-001d877344eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" Nov 03 20:32:52.095174 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:52.095138 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sztw8\" (UniqueName: \"kubernetes.io/projected/6b991442-a874-43f1-bf91-001d877344eb-kube-api-access-sztw8\") pod \"maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd\" (UID: \"6b991442-a874-43f1-bf91-001d877344eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" Nov 03 20:32:52.231732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:52.231647 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" Nov 03 20:32:52.387028 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:52.386993 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5"] Nov 03 20:32:52.419408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:52.418329 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9"] Nov 03 20:32:52.545729 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:32:52.545698 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b991442_a874_43f1_bf91_001d877344eb.slice/crio-9b93f4e4f5bd63913544611be93713d2edf824ed6b374b709cfbd63743625ed6 WatchSource:0}: Error finding container 9b93f4e4f5bd63913544611be93713d2edf824ed6b374b709cfbd63743625ed6: Status 404 returned error can't find the container with id 9b93f4e4f5bd63913544611be93713d2edf824ed6b374b709cfbd63743625ed6 Nov 03 20:32:52.547532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:52.547510 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd"] Nov 03 20:32:53.444231 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:53.444132 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" event={"ID":"6b991442-a874-43f1-bf91-001d877344eb","Type":"ContainerStarted","Data":"2e979c827788d2855cfd94104f3b6c69351fbf3e609bb71d83d51dc242456283"} Nov 03 20:32:53.444829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:53.444777 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" event={"ID":"6b991442-a874-43f1-bf91-001d877344eb","Type":"ContainerStarted","Data":"9b93f4e4f5bd63913544611be93713d2edf824ed6b374b709cfbd63743625ed6"} Nov 03 20:32:53.470493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:53.469296 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" podStartSLOduration=2.469281354 podStartE2EDuration="2.469281354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:32:53.465219839 +0000 UTC m=+2151.840816586" watchObservedRunningTime="2026-11-03 20:32:53.469281354 +0000 UTC m=+2151.844878100" Nov 03 20:32:53.531525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:53.531481 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" podUID="9deb1243-b43c-4e37-94a9-1eb5e96c21cc" containerName="payload-processing" containerID="cri-o://9f3e9cf7c85c34eab1eb05a24c462671d4f5414f348d682ef5595d83e6572195" gracePeriod=25 Nov 03 20:32:53.540988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:53.540955 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" podUID="7f934e95-c712-48af-83d4-a7b80d870089" containerName="payload-pre-processing" containerID="cri-o://769b176777ba6c1539ce2059ff6006c048d1e64dc330e477bb939645bb5ad55f" gracePeriod=25 Nov 03 20:32:54.129314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.129288 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" Nov 03 20:32:54.225077 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.225037 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9899751b-58b7-428e-a0b1-464a005dfabe-kube-api-access-7sm4b\" (UniqueName: \"kubernetes.io/projected/9899751b-58b7-428e-a0b1-464a005dfabe-kube-api-access-7sm4b\") pod \"9899751b-58b7-428e-a0b1-464a005dfabe\" (UID: \"9899751b-58b7-428e-a0b1-464a005dfabe\") " Nov 03 20:32:54.225284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.225162 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9899751b-58b7-428e-a0b1-464a005dfabe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899751b-58b7-428e-a0b1-464a005dfabe-plugins-config-volume\") pod \"9899751b-58b7-428e-a0b1-464a005dfabe\" (UID: \"9899751b-58b7-428e-a0b1-464a005dfabe\") " Nov 03 20:32:54.225630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.225599 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9899751b-58b7-428e-a0b1-464a005dfabe-plugins-config-volume" pod "9899751b-58b7-428e-a0b1-464a005dfabe" (UID: "9899751b-58b7-428e-a0b1-464a005dfabe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:54.226034 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.226012 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9899751b-58b7-428e-a0b1-464a005dfabe-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:54.227449 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.227371 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9899751b-58b7-428e-a0b1-464a005dfabe-kube-api-access-7sm4b" pod "9899751b-58b7-428e-a0b1-464a005dfabe" (UID: "9899751b-58b7-428e-a0b1-464a005dfabe"). InnerVolumeSpecName "kube-api-access-7sm4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:54.327232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.327195 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sm4b\" (UniqueName: \"kubernetes.io/projected/9899751b-58b7-428e-a0b1-464a005dfabe-kube-api-access-7sm4b\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:54.350240 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.350157 4457 patch_prober.go:29] interesting pod/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:32:54.350240 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.350215 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" podUID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:32:54.455275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.455237 4457 generic.go:356] "Generic (PLEG): container finished" podID="9899751b-58b7-428e-a0b1-464a005dfabe" containerID="863ec92fefd1eb2585e90128af0770cfe26667573146e522acb5356a215cdf7d" exitCode=0 Nov 03 20:32:54.455774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.455290 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" Nov 03 20:32:54.455774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.455317 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" event={"ID":"9899751b-58b7-428e-a0b1-464a005dfabe","Type":"ContainerDied","Data":"863ec92fefd1eb2585e90128af0770cfe26667573146e522acb5356a215cdf7d"} Nov 03 20:32:54.455774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.455377 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j" event={"ID":"9899751b-58b7-428e-a0b1-464a005dfabe","Type":"ContainerDied","Data":"4f7e8a307327f8aec6f8c87492386c9390adfd42ce75ae608cc79a8da9290d9d"} Nov 03 20:32:54.455774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.455424 4457 scope.go:122] "RemoveContainer" containerID="863ec92fefd1eb2585e90128af0770cfe26667573146e522acb5356a215cdf7d" Nov 03 20:32:54.457921 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.457895 4457 generic.go:356] "Generic (PLEG): container finished" podID="6b991442-a874-43f1-bf91-001d877344eb" containerID="2e979c827788d2855cfd94104f3b6c69351fbf3e609bb71d83d51dc242456283" exitCode=0 Nov 03 20:32:54.458227 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.457933 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" event={"ID":"6b991442-a874-43f1-bf91-001d877344eb","Type":"ContainerDied","Data":"2e979c827788d2855cfd94104f3b6c69351fbf3e609bb71d83d51dc242456283"} Nov 03 20:32:54.473553 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.473528 4457 scope.go:122] "RemoveContainer" containerID="863ec92fefd1eb2585e90128af0770cfe26667573146e522acb5356a215cdf7d" Nov 03 20:32:54.473934 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:32:54.473909 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863ec92fefd1eb2585e90128af0770cfe26667573146e522acb5356a215cdf7d\": container with ID starting with 863ec92fefd1eb2585e90128af0770cfe26667573146e522acb5356a215cdf7d not found: ID does not exist" containerID="863ec92fefd1eb2585e90128af0770cfe26667573146e522acb5356a215cdf7d" Nov 03 20:32:54.474055 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.473941 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863ec92fefd1eb2585e90128af0770cfe26667573146e522acb5356a215cdf7d"} err="failed to get container status \"863ec92fefd1eb2585e90128af0770cfe26667573146e522acb5356a215cdf7d\": rpc error: code = NotFound desc = could not find container \"863ec92fefd1eb2585e90128af0770cfe26667573146e522acb5356a215cdf7d\": container with ID starting with 863ec92fefd1eb2585e90128af0770cfe26667573146e522acb5356a215cdf7d not found: ID does not exist" Nov 03 20:32:54.501716 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.501683 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j"] Nov 03 20:32:54.507182 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:54.507153 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-34e844-6c7c48d787-z2n4j"] Nov 03 20:32:55.712603 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:55.712575 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" Nov 03 20:32:55.769143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:55.769061 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9899751b-58b7-428e-a0b1-464a005dfabe" path="/var/lib/kubelet/pods/9899751b-58b7-428e-a0b1-464a005dfabe/volumes" Nov 03 20:32:55.861592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:55.861560 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b991442-a874-43f1-bf91-001d877344eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b991442-a874-43f1-bf91-001d877344eb-maas-api-xks-ca\") pod \"6b991442-a874-43f1-bf91-001d877344eb\" (UID: \"6b991442-a874-43f1-bf91-001d877344eb\") " Nov 03 20:32:55.861780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:55.861650 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b991442-a874-43f1-bf91-001d877344eb-kube-api-access-sztw8\" (UniqueName: \"kubernetes.io/projected/6b991442-a874-43f1-bf91-001d877344eb-kube-api-access-sztw8\") pod \"6b991442-a874-43f1-bf91-001d877344eb\" (UID: \"6b991442-a874-43f1-bf91-001d877344eb\") " Nov 03 20:32:55.861947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:55.861926 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b991442-a874-43f1-bf91-001d877344eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b991442-a874-43f1-bf91-001d877344eb-maas-api-openshift-ca\") pod \"6b991442-a874-43f1-bf91-001d877344eb\" (UID: \"6b991442-a874-43f1-bf91-001d877344eb\") " Nov 03 20:32:55.862327 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:55.862303 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b991442-a874-43f1-bf91-001d877344eb-maas-api-openshift-ca" pod "6b991442-a874-43f1-bf91-001d877344eb" (UID: "6b991442-a874-43f1-bf91-001d877344eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:55.862615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:55.862596 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b991442-a874-43f1-bf91-001d877344eb-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:55.864252 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:55.864220 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b991442-a874-43f1-bf91-001d877344eb-maas-api-xks-ca" pod "6b991442-a874-43f1-bf91-001d877344eb" (UID: "6b991442-a874-43f1-bf91-001d877344eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:32:55.864343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:55.864274 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b991442-a874-43f1-bf91-001d877344eb-kube-api-access-sztw8" pod "6b991442-a874-43f1-bf91-001d877344eb" (UID: "6b991442-a874-43f1-bf91-001d877344eb"). InnerVolumeSpecName "kube-api-access-sztw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:55.964105 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:55.964068 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b991442-a874-43f1-bf91-001d877344eb-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:55.964105 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:55.964106 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sztw8\" (UniqueName: \"kubernetes.io/projected/6b991442-a874-43f1-bf91-001d877344eb-kube-api-access-sztw8\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.512808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.511891 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" Nov 03 20:32:56.512808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.511926 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd" event={"ID":"6b991442-a874-43f1-bf91-001d877344eb","Type":"ContainerDied","Data":"9b93f4e4f5bd63913544611be93713d2edf824ed6b374b709cfbd63743625ed6"} Nov 03 20:32:56.512808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.511974 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b93f4e4f5bd63913544611be93713d2edf824ed6b374b709cfbd63743625ed6" Nov 03 20:32:56.515913 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.515886 4457 generic.go:356] "Generic (PLEG): container finished" podID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" containerID="4dd34ca74dc2ca470921cdaaf15561ce5bbaebac9ff00968b6b051ec23e115eb" exitCode=0 Nov 03 20:32:56.516236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.516070 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" event={"ID":"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22","Type":"ContainerDied","Data":"4dd34ca74dc2ca470921cdaaf15561ce5bbaebac9ff00968b6b051ec23e115eb"} Nov 03 20:32:56.667797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.667771 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:32:56.722930 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.722874 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l"] Nov 03 20:32:56.748693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.748031 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8"] Nov 03 20:32:56.787605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.787440 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9"] Nov 03 20:32:56.813946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.813586 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-envoy\") pod \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " Nov 03 20:32:56.813946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.813665 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istiod-ca-cert\") pod \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " Nov 03 20:32:56.813946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.813705 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-socket\") pod \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " Nov 03 20:32:56.813946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.813780 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-token\" (UniqueName: \"kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-token\") pod \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " Nov 03 20:32:56.813946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.813811 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-kube-api-access-f9fkm\" (UniqueName: \"kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-kube-api-access-f9fkm\") pod \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " Nov 03 20:32:56.814373 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.814075 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-data\") pod \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " Nov 03 20:32:56.814373 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.814161 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-podinfo\") pod \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " Nov 03 20:32:56.814373 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.814200 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-credential-socket\") pod \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " Nov 03 20:32:56.814373 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.814267 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-certs\") pod \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\" (UID: \"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22\") " Nov 03 20:32:56.815265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.815237 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-certs" pod "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" (UID: "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:56.816453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.816313 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-credential-socket" pod "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" (UID: "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:56.816779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.816596 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-socket" pod "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" (UID: "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:56.816961 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.816913 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-data" pod "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" (UID: "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:56.817233 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.817194 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istiod-ca-cert" pod "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" (UID: "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:56.829694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.829651 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-podinfo" pod "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" (UID: "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:56.831196 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.831164 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-kube-api-access-f9fkm" pod "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" (UID: "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22"). InnerVolumeSpecName "kube-api-access-f9fkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:56.832776 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.832729 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-envoy" pod "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" (UID: "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:56.833206 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.833180 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-token" pod "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" (UID: "117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:56.918987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.918956 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.919222 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.919206 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.919319 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.919307 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.919463 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.919449 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.919615 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.919598 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9fkm\" (UniqueName: \"kubernetes.io/projected/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-kube-api-access-f9fkm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.919733 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.919720 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.919832 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.919818 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.919926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.919913 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:56.920024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:56.920011 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.139412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.106059 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:32:57.139412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.134343 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-podinfo\") pod \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " Nov 03 20:32:57.139412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.134490 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-kube-api-access-t4q52\" (UniqueName: \"kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-kube-api-access-t4q52\") pod \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " Nov 03 20:32:57.139412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.134562 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-socket\") pod \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " Nov 03 20:32:57.151943 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.151905 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-socket" pod "e8bcbefd-88d8-4eae-9cf5-fa29620724a2" (UID: "e8bcbefd-88d8-4eae-9cf5-fa29620724a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:57.152940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.152913 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-envoy\") pod \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " Nov 03 20:32:57.153165 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.153148 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-data\") pod \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " Nov 03 20:32:57.153307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.153291 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istiod-ca-cert\") pod \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " Nov 03 20:32:57.153520 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.153499 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-certs\") pod \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " Nov 03 20:32:57.153650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.153635 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-token\" (UniqueName: \"kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-token\") pod \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " Nov 03 20:32:57.153758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.153743 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-credential-socket\") pod \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\" (UID: \"e8bcbefd-88d8-4eae-9cf5-fa29620724a2\") " Nov 03 20:32:57.159430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.159136 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-certs" pod "e8bcbefd-88d8-4eae-9cf5-fa29620724a2" (UID: "e8bcbefd-88d8-4eae-9cf5-fa29620724a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:57.167493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.167464 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-data" pod "e8bcbefd-88d8-4eae-9cf5-fa29620724a2" (UID: "e8bcbefd-88d8-4eae-9cf5-fa29620724a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:57.178409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.171637 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-credential-socket" pod "e8bcbefd-88d8-4eae-9cf5-fa29620724a2" (UID: "e8bcbefd-88d8-4eae-9cf5-fa29620724a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:57.178409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.171993 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.178409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.172012 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.178409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.172025 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.178409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.172049 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.178409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.173592 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istiod-ca-cert" pod "e8bcbefd-88d8-4eae-9cf5-fa29620724a2" (UID: "e8bcbefd-88d8-4eae-9cf5-fa29620724a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:32:57.203004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.202948 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-token" pod "e8bcbefd-88d8-4eae-9cf5-fa29620724a2" (UID: "e8bcbefd-88d8-4eae-9cf5-fa29620724a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:57.203425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.203370 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-kube-api-access-t4q52" pod "e8bcbefd-88d8-4eae-9cf5-fa29620724a2" (UID: "e8bcbefd-88d8-4eae-9cf5-fa29620724a2"). InnerVolumeSpecName "kube-api-access-t4q52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:32:57.222425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.222363 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-podinfo" pod "e8bcbefd-88d8-4eae-9cf5-fa29620724a2" (UID: "e8bcbefd-88d8-4eae-9cf5-fa29620724a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:32:57.223095 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.223054 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-envoy" pod "e8bcbefd-88d8-4eae-9cf5-fa29620724a2" (UID: "e8bcbefd-88d8-4eae-9cf5-fa29620724a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:32:57.273605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.273499 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.273605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.273535 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.273605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.273549 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.273605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.273561 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.273605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.273573 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4q52\" (UniqueName: \"kubernetes.io/projected/e8bcbefd-88d8-4eae-9cf5-fa29620724a2-kube-api-access-t4q52\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:32:57.388318 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.387292 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" podUID="6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7" containerName="payload-processing" containerID="cri-o://4d051ef325f777b6f42b652fc23c61da1dadc49e790c42debe1211bed9e1700f" gracePeriod=25 Nov 03 20:32:57.419363 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.419205 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" podUID="fad0ed4c-3ba5-4a1c-a221-668576f5d27e" containerName="payload-pre-processing" containerID="cri-o://91d322d9a065425bc4e4284396cc9c8878aa58ac59c9e4ce80bd6a27ed5503d5" gracePeriod=25 Nov 03 20:32:57.445957 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.445922 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq"] Nov 03 20:32:57.453974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.453934 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7ae0f1-qprrq"] Nov 03 20:32:57.534428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.526832 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" Nov 03 20:32:57.534428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.526870 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk" event={"ID":"117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22","Type":"ContainerDied","Data":"94eae0847f62f13493081e5726f28f8dae6c4fac43b4a6bf5df57ce12fe098a3"} Nov 03 20:32:57.534428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.526937 4457 scope.go:122] "RemoveContainer" containerID="4dd34ca74dc2ca470921cdaaf15561ce5bbaebac9ff00968b6b051ec23e115eb" Nov 03 20:32:57.534428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.528817 4457 generic.go:356] "Generic (PLEG): container finished" podID="e8bcbefd-88d8-4eae-9cf5-fa29620724a2" containerID="65b023e3e9738b2de64fd05f238c8b0e5bdb2efca4f00efd644f6af140f51850" exitCode=0 Nov 03 20:32:57.534428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.528874 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" Nov 03 20:32:57.534428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.528944 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" event={"ID":"e8bcbefd-88d8-4eae-9cf5-fa29620724a2","Type":"ContainerDied","Data":"65b023e3e9738b2de64fd05f238c8b0e5bdb2efca4f00efd644f6af140f51850"} Nov 03 20:32:57.534428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.528995 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6" event={"ID":"e8bcbefd-88d8-4eae-9cf5-fa29620724a2","Type":"ContainerDied","Data":"17ce3395795541d50bd6a283acc6baf77c80fe20b1967060e575b3d27682c594"} Nov 03 20:32:57.541502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.541446 4457 scope.go:122] "RemoveContainer" containerID="65b023e3e9738b2de64fd05f238c8b0e5bdb2efca4f00efd644f6af140f51850" Nov 03 20:32:57.578132 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.578005 4457 scope.go:122] "RemoveContainer" containerID="65b023e3e9738b2de64fd05f238c8b0e5bdb2efca4f00efd644f6af140f51850" Nov 03 20:32:57.589531 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:32:57.584790 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b023e3e9738b2de64fd05f238c8b0e5bdb2efca4f00efd644f6af140f51850\": container with ID starting with 65b023e3e9738b2de64fd05f238c8b0e5bdb2efca4f00efd644f6af140f51850 not found: ID does not exist" containerID="65b023e3e9738b2de64fd05f238c8b0e5bdb2efca4f00efd644f6af140f51850" Nov 03 20:32:57.589531 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.584837 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b023e3e9738b2de64fd05f238c8b0e5bdb2efca4f00efd644f6af140f51850"} err="failed to get container status \"65b023e3e9738b2de64fd05f238c8b0e5bdb2efca4f00efd644f6af140f51850\": rpc error: code = NotFound desc = could not find container \"65b023e3e9738b2de64fd05f238c8b0e5bdb2efca4f00efd644f6af140f51850\": container with ID starting with 65b023e3e9738b2de64fd05f238c8b0e5bdb2efca4f00efd644f6af140f51850 not found: ID does not exist" Nov 03 20:32:57.626634 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.626599 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk"] Nov 03 20:32:57.634052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.634022 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-34e844-openshift-default-7775789977-gqrdk"] Nov 03 20:32:57.639647 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.639624 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6"] Nov 03 20:32:57.644481 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.644461 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6cf8ae-openshift-default-578b8dd5cc-9hrf6"] Nov 03 20:32:57.762795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.762763 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" path="/var/lib/kubelet/pods/117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22/volumes" Nov 03 20:32:57.763465 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.763444 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17905aef-859b-48d6-8bdc-46525adea158" path="/var/lib/kubelet/pods/17905aef-859b-48d6-8bdc-46525adea158/volumes" Nov 03 20:32:57.764121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:57.764103 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8bcbefd-88d8-4eae-9cf5-fa29620724a2" path="/var/lib/kubelet/pods/e8bcbefd-88d8-4eae-9cf5-fa29620724a2/volumes" Nov 03 20:32:58.529411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:58.523129 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" podUID="fad0ed4c-3ba5-4a1c-a221-668576f5d27e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:32:58.529411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:58.524952 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" podUID="6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:32:58.679486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:58.679445 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf"] Nov 03 20:32:58.694407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:58.688796 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-6cf8ae-zlspf"] Nov 03 20:32:59.763345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:32:59.763306 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78469a88-9d85-4e50-bec8-efdb5fdc1a5e" path="/var/lib/kubelet/pods/78469a88-9d85-4e50-bec8-efdb5fdc1a5e/volumes" Nov 03 20:33:00.760199 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:00.760151 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" podUID="2699c9a3-baf0-40f0-b1be-892b29a62ad4" containerName="main" containerID="cri-o://8f849a8f2bc84270ebd0632220f6673d758c31db2320ffbbf01ccc9807cf0bf9" gracePeriod=45 Nov 03 20:33:01.452087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.452055 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:33:01.639855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.639528 4457 generic.go:356] "Generic (PLEG): container finished" podID="2699c9a3-baf0-40f0-b1be-892b29a62ad4" containerID="8f849a8f2bc84270ebd0632220f6673d758c31db2320ffbbf01ccc9807cf0bf9" exitCode=0 Nov 03 20:33:01.639855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.639579 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" event={"ID":"2699c9a3-baf0-40f0-b1be-892b29a62ad4","Type":"ContainerDied","Data":"8f849a8f2bc84270ebd0632220f6673d758c31db2320ffbbf01ccc9807cf0bf9"} Nov 03 20:33:01.639855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.639603 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" Nov 03 20:33:01.639855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.639642 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" event={"ID":"2699c9a3-baf0-40f0-b1be-892b29a62ad4","Type":"ContainerDied","Data":"02a7bf7cf3332a3e2d7cacf1e9d91d4bf084aac59f6d5c98f777532411c6d531"} Nov 03 20:33:01.639855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.639668 4457 scope.go:122] "RemoveContainer" containerID="8f849a8f2bc84270ebd0632220f6673d758c31db2320ffbbf01ccc9807cf0bf9" Nov 03 20:33:01.643742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.643715 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-home\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-home\") pod \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " Nov 03 20:33:01.643879 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.643773 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-dshm\") pod \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " Nov 03 20:33:01.643944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.643903 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tmp-dir\") pod \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " Nov 03 20:33:01.644002 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.643973 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-home" pod "2699c9a3-baf0-40f0-b1be-892b29a62ad4" (UID: "2699c9a3-baf0-40f0-b1be-892b29a62ad4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:01.644062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.644003 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-model-cache\") pod \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " Nov 03 20:33:01.644062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.644035 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tls-certs\" (UniqueName: \"kubernetes.io/secret/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tls-certs\") pod \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " Nov 03 20:33:01.644179 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.644161 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2699c9a3-baf0-40f0-b1be-892b29a62ad4-kube-api-access-r4jbk\" (UniqueName: \"kubernetes.io/projected/2699c9a3-baf0-40f0-b1be-892b29a62ad4-kube-api-access-r4jbk\") pod \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\" (UID: \"2699c9a3-baf0-40f0-b1be-892b29a62ad4\") " Nov 03 20:33:01.644233 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.644168 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tmp-dir" pod "2699c9a3-baf0-40f0-b1be-892b29a62ad4" (UID: "2699c9a3-baf0-40f0-b1be-892b29a62ad4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:01.644338 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.644240 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-model-cache" pod "2699c9a3-baf0-40f0-b1be-892b29a62ad4" (UID: "2699c9a3-baf0-40f0-b1be-892b29a62ad4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:01.644860 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.644835 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:01.645042 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.644862 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:01.645042 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.644903 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:01.646352 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.646329 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tls-certs" pod "2699c9a3-baf0-40f0-b1be-892b29a62ad4" (UID: "2699c9a3-baf0-40f0-b1be-892b29a62ad4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:01.646629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.646605 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2699c9a3-baf0-40f0-b1be-892b29a62ad4-kube-api-access-r4jbk" pod "2699c9a3-baf0-40f0-b1be-892b29a62ad4" (UID: "2699c9a3-baf0-40f0-b1be-892b29a62ad4"). InnerVolumeSpecName "kube-api-access-r4jbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:01.646727 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.646643 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-dshm" pod "2699c9a3-baf0-40f0-b1be-892b29a62ad4" (UID: "2699c9a3-baf0-40f0-b1be-892b29a62ad4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:01.653760 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.653741 4457 scope.go:122] "RemoveContainer" containerID="8f849a8f2bc84270ebd0632220f6673d758c31db2320ffbbf01ccc9807cf0bf9" Nov 03 20:33:01.654057 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:01.654037 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f849a8f2bc84270ebd0632220f6673d758c31db2320ffbbf01ccc9807cf0bf9\": container with ID starting with 8f849a8f2bc84270ebd0632220f6673d758c31db2320ffbbf01ccc9807cf0bf9 not found: ID does not exist" containerID="8f849a8f2bc84270ebd0632220f6673d758c31db2320ffbbf01ccc9807cf0bf9" Nov 03 20:33:01.654128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.654064 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f849a8f2bc84270ebd0632220f6673d758c31db2320ffbbf01ccc9807cf0bf9"} err="failed to get container status \"8f849a8f2bc84270ebd0632220f6673d758c31db2320ffbbf01ccc9807cf0bf9\": rpc error: code = NotFound desc = could not find container \"8f849a8f2bc84270ebd0632220f6673d758c31db2320ffbbf01ccc9807cf0bf9\": container with ID starting with 8f849a8f2bc84270ebd0632220f6673d758c31db2320ffbbf01ccc9807cf0bf9 not found: ID does not exist" Nov 03 20:33:01.746079 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.746045 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2699c9a3-baf0-40f0-b1be-892b29a62ad4-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:01.746079 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.746076 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4jbk\" (UniqueName: \"kubernetes.io/projected/2699c9a3-baf0-40f0-b1be-892b29a62ad4-kube-api-access-r4jbk\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:01.746079 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.746089 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2699c9a3-baf0-40f0-b1be-892b29a62ad4-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:01.960540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.960455 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56"] Nov 03 20:33:01.965574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:01.965543 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56"] Nov 03 20:33:02.292501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:02.277877 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-b0adda/test-model-b0adda-kserve-65d7cc555-ddt56" podUID="2699c9a3-baf0-40f0-b1be-892b29a62ad4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.194:8000/ready\": context deadline exceeded" Nov 03 20:33:02.704519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:02.693125 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" podUID="7f934e95-c712-48af-83d4-a7b80d870089" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:02.704519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:02.699998 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" podUID="9deb1243-b43c-4e37-94a9-1eb5e96c21cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:03.766863 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:03.766827 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2699c9a3-baf0-40f0-b1be-892b29a62ad4" path="/var/lib/kubelet/pods/2699c9a3-baf0-40f0-b1be-892b29a62ad4/volumes" Nov 03 20:33:04.439990 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:04.439707 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq"] Nov 03 20:33:04.440197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:04.439978 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" podUID="f67ad7d6-ee78-41e7-9644-81e1989198c1" containerName="maas-api" containerID="cri-o://ee0406e57cda331a45f07c4961756e81a25ffcfa7fd3eb3edd8b8dc67e6e1d18" gracePeriod=30 Nov 03 20:33:04.922234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:04.921891 4457 generic.go:356] "Generic (PLEG): container finished" podID="f67ad7d6-ee78-41e7-9644-81e1989198c1" containerID="ee0406e57cda331a45f07c4961756e81a25ffcfa7fd3eb3edd8b8dc67e6e1d18" exitCode=0 Nov 03 20:33:04.922234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:04.922147 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" event={"ID":"f67ad7d6-ee78-41e7-9644-81e1989198c1","Type":"ContainerDied","Data":"ee0406e57cda331a45f07c4961756e81a25ffcfa7fd3eb3edd8b8dc67e6e1d18"} Nov 03 20:33:05.038744 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.038710 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b"] Nov 03 20:33:05.070337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.070305 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8c684d-bfk7b"] Nov 03 20:33:05.102484 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.102452 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:33:05.155528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.155490 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67ad7d6-ee78-41e7-9644-81e1989198c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67ad7d6-ee78-41e7-9644-81e1989198c1-openshift-service-ca\") pod \"f67ad7d6-ee78-41e7-9644-81e1989198c1\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " Nov 03 20:33:05.155738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.155561 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-maas-api-tls\") pod \"f67ad7d6-ee78-41e7-9644-81e1989198c1\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " Nov 03 20:33:05.155738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.155660 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67ad7d6-ee78-41e7-9644-81e1989198c1-kube-api-access-zcwbz\" (UniqueName: \"kubernetes.io/projected/f67ad7d6-ee78-41e7-9644-81e1989198c1-kube-api-access-zcwbz\") pod \"f67ad7d6-ee78-41e7-9644-81e1989198c1\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " Nov 03 20:33:05.155849 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.155816 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-metrics-certs\") pod \"f67ad7d6-ee78-41e7-9644-81e1989198c1\" (UID: \"f67ad7d6-ee78-41e7-9644-81e1989198c1\") " Nov 03 20:33:05.155903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.155881 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67ad7d6-ee78-41e7-9644-81e1989198c1-openshift-service-ca" pod "f67ad7d6-ee78-41e7-9644-81e1989198c1" (UID: "f67ad7d6-ee78-41e7-9644-81e1989198c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:05.157229 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.157178 4457 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f67ad7d6-ee78-41e7-9644-81e1989198c1-openshift-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:05.166383 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.158904 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-maas-api-tls" pod "f67ad7d6-ee78-41e7-9644-81e1989198c1" (UID: "f67ad7d6-ee78-41e7-9644-81e1989198c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:05.166383 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.159942 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-metrics-certs" pod "f67ad7d6-ee78-41e7-9644-81e1989198c1" (UID: "f67ad7d6-ee78-41e7-9644-81e1989198c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:05.166383 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.160827 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67ad7d6-ee78-41e7-9644-81e1989198c1-kube-api-access-zcwbz" pod "f67ad7d6-ee78-41e7-9644-81e1989198c1" (UID: "f67ad7d6-ee78-41e7-9644-81e1989198c1"). InnerVolumeSpecName "kube-api-access-zcwbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:05.259102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.259059 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-maas-api-tls\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:05.259102 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.259101 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcwbz\" (UniqueName: \"kubernetes.io/projected/f67ad7d6-ee78-41e7-9644-81e1989198c1-kube-api-access-zcwbz\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:05.259379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.259118 4457 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f67ad7d6-ee78-41e7-9644-81e1989198c1-metrics-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:05.777957 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.777914 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a618b730-ae49-42ce-ae44-aabfdd1a19cf" path="/var/lib/kubelet/pods/a618b730-ae49-42ce-ae44-aabfdd1a19cf/volumes" Nov 03 20:33:05.925538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.925496 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22"] Nov 03 20:33:05.927198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.927157 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" podUID="8d108227-cd53-4d58-9d65-1ff5fd6d1149" containerName="istio-proxy" containerID="cri-o://644b7684c182052a788c5daf6815a0d00a650b751249db5a11830f58e16e8db2" gracePeriod=30 Nov 03 20:33:05.944037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.944002 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" Nov 03 20:33:05.944037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.944014 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq" event={"ID":"f67ad7d6-ee78-41e7-9644-81e1989198c1","Type":"ContainerDied","Data":"5b0ff6493437aac5b1f118fdcc66b7a87a47d6979eb473dffd30a0c023d2e975"} Nov 03 20:33:05.944237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.944093 4457 scope.go:122] "RemoveContainer" containerID="ee0406e57cda331a45f07c4961756e81a25ffcfa7fd3eb3edd8b8dc67e6e1d18" Nov 03 20:33:05.981760 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.981722 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq"] Nov 03 20:33:05.986691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:05.986666 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8c684d-8448bb5667-4l8dq"] Nov 03 20:33:06.208819 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:06.208783 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:33:06.366953 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.366914 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k"] Nov 03 20:33:06.368101 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368071 4457 cpu_manager.go:414] "containerMap: removing container" podUID="9899751b-58b7-428e-a0b1-464a005dfabe" containerName="payload-pre-processing" Nov 03 20:33:06.368101 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368102 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9899751b-58b7-428e-a0b1-464a005dfabe" containerName="payload-pre-processing" Nov 03 20:33:06.368278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368135 4457 cpu_manager.go:414] "containerMap: removing container" podUID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" containerName="istio-proxy" Nov 03 20:33:06.368278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368142 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" containerName="istio-proxy" Nov 03 20:33:06.368278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368171 4457 cpu_manager.go:414] "containerMap: removing container" podUID="6b991442-a874-43f1-bf91-001d877344eb" containerName="revoke-keys" Nov 03 20:33:06.368278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368178 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b991442-a874-43f1-bf91-001d877344eb" containerName="revoke-keys" Nov 03 20:33:06.368278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368198 4457 cpu_manager.go:414] "containerMap: removing container" podUID="e8bcbefd-88d8-4eae-9cf5-fa29620724a2" containerName="istio-proxy" Nov 03 20:33:06.368278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368205 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bcbefd-88d8-4eae-9cf5-fa29620724a2" containerName="istio-proxy" Nov 03 20:33:06.368278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368217 4457 cpu_manager.go:414] "containerMap: removing container" podUID="f67ad7d6-ee78-41e7-9644-81e1989198c1" containerName="maas-api" Nov 03 20:33:06.368278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368225 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67ad7d6-ee78-41e7-9644-81e1989198c1" containerName="maas-api" Nov 03 20:33:06.368278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368236 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2699c9a3-baf0-40f0-b1be-892b29a62ad4" containerName="main" Nov 03 20:33:06.368278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368243 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2699c9a3-baf0-40f0-b1be-892b29a62ad4" containerName="main" Nov 03 20:33:06.368926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368470 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b991442-a874-43f1-bf91-001d877344eb" containerName="revoke-keys" Nov 03 20:33:06.368926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368491 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bcbefd-88d8-4eae-9cf5-fa29620724a2" containerName="istio-proxy" Nov 03 20:33:06.368926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368503 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67ad7d6-ee78-41e7-9644-81e1989198c1" containerName="maas-api" Nov 03 20:33:06.368926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368512 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="2699c9a3-baf0-40f0-b1be-892b29a62ad4" containerName="main" Nov 03 20:33:06.368926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368531 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="9899751b-58b7-428e-a0b1-464a005dfabe" containerName="payload-pre-processing" Nov 03 20:33:06.368926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.368545 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="117fccf6-6e71-49c1-a9d0-d5f0c4d6eb22" containerName="istio-proxy" Nov 03 20:33:06.375040 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.375009 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" Nov 03 20:33:06.379019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.378994 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:33:06.383636 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.383552 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:33:06.387173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.387143 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k"] Nov 03 20:33:06.490157 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.490053 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d0898b-8847k\" (UID: \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" Nov 03 20:33:06.490351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.490159 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d0898b-8847k\" (UID: \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" Nov 03 20:33:06.490351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.490191 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mc2j\" (UniqueName: \"kubernetes.io/projected/fdaa8957-b04b-43f6-8ee4-27212c7f9840-kube-api-access-6mc2j\") pod \"maas-api-revoke-keys-e2e-inf-b-d0898b-8847k\" (UID: \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" Nov 03 20:33:06.591763 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.591724 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d0898b-8847k\" (UID: \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" Nov 03 20:33:06.591948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.591792 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d0898b-8847k\" (UID: \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" Nov 03 20:33:06.591948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.591821 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mc2j\" (UniqueName: \"kubernetes.io/projected/fdaa8957-b04b-43f6-8ee4-27212c7f9840-kube-api-access-6mc2j\") pod \"maas-api-revoke-keys-e2e-inf-b-d0898b-8847k\" (UID: \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" Nov 03 20:33:06.592613 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.592581 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d0898b-8847k\" (UID: \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" Nov 03 20:33:06.596067 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.594715 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d0898b-8847k\" (UID: \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" Nov 03 20:33:06.610456 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.610425 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mc2j\" (UniqueName: \"kubernetes.io/projected/fdaa8957-b04b-43f6-8ee4-27212c7f9840-kube-api-access-6mc2j\") pod \"maas-api-revoke-keys-e2e-inf-b-d0898b-8847k\" (UID: \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" Nov 03 20:33:06.715651 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:06.715609 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" Nov 03 20:33:07.179556 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.179491 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k"] Nov 03 20:33:07.179556 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:33:07.179523 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdaa8957_b04b_43f6_8ee4_27212c7f9840.slice/crio-66e20453cfb55455bac4eec7ef4215d6de4bda197a4fcb8f20358e9d9cfc75b4 WatchSource:0}: Error finding container 66e20453cfb55455bac4eec7ef4215d6de4bda197a4fcb8f20358e9d9cfc75b4: Status 404 returned error can't find the container with id 66e20453cfb55455bac4eec7ef4215d6de4bda197a4fcb8f20358e9d9cfc75b4 Nov 03 20:33:07.195951 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.195926 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl"] Nov 03 20:33:07.203122 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.203043 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" Nov 03 20:33:07.208382 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.208352 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl"] Nov 03 20:33:07.323475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.323399 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl\" (UID: \"435839a4-9123-4b35-83d7-aaa4103ccc28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" Nov 03 20:33:07.323653 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.323632 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kwtb\" (UniqueName: \"kubernetes.io/projected/435839a4-9123-4b35-83d7-aaa4103ccc28-kube-api-access-9kwtb\") pod \"maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl\" (UID: \"435839a4-9123-4b35-83d7-aaa4103ccc28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" Nov 03 20:33:07.323795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.323775 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl\" (UID: \"435839a4-9123-4b35-83d7-aaa4103ccc28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" Nov 03 20:33:07.424971 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.424932 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl\" (UID: \"435839a4-9123-4b35-83d7-aaa4103ccc28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" Nov 03 20:33:07.425127 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.425074 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kwtb\" (UniqueName: \"kubernetes.io/projected/435839a4-9123-4b35-83d7-aaa4103ccc28-kube-api-access-9kwtb\") pod \"maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl\" (UID: \"435839a4-9123-4b35-83d7-aaa4103ccc28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" Nov 03 20:33:07.425204 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.425170 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl\" (UID: \"435839a4-9123-4b35-83d7-aaa4103ccc28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" Nov 03 20:33:07.425907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.425880 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl\" (UID: \"435839a4-9123-4b35-83d7-aaa4103ccc28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" Nov 03 20:33:07.427459 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.427440 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl\" (UID: \"435839a4-9123-4b35-83d7-aaa4103ccc28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" Nov 03 20:33:07.437271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.437248 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kwtb\" (UniqueName: \"kubernetes.io/projected/435839a4-9123-4b35-83d7-aaa4103ccc28-kube-api-access-9kwtb\") pod \"maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl\" (UID: \"435839a4-9123-4b35-83d7-aaa4103ccc28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" Nov 03 20:33:07.580773 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.580745 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" Nov 03 20:33:07.774359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.774321 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67ad7d6-ee78-41e7-9644-81e1989198c1" path="/var/lib/kubelet/pods/f67ad7d6-ee78-41e7-9644-81e1989198c1/volumes" Nov 03 20:33:07.926606 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:33:07.926566 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod435839a4_9123_4b35_83d7_aaa4103ccc28.slice/crio-56bb7703b9c948ffc1aa236168a8b06186eb0061c0faf536fe5971e4db93c429 WatchSource:0}: Error finding container 56bb7703b9c948ffc1aa236168a8b06186eb0061c0faf536fe5971e4db93c429: Status 404 returned error can't find the container with id 56bb7703b9c948ffc1aa236168a8b06186eb0061c0faf536fe5971e4db93c429 Nov 03 20:33:07.928365 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.928335 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl"] Nov 03 20:33:07.997811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.997773 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" event={"ID":"435839a4-9123-4b35-83d7-aaa4103ccc28","Type":"ContainerStarted","Data":"56bb7703b9c948ffc1aa236168a8b06186eb0061c0faf536fe5971e4db93c429"} Nov 03 20:33:07.999852 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.999822 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" event={"ID":"fdaa8957-b04b-43f6-8ee4-27212c7f9840","Type":"ContainerStarted","Data":"8139269e799af25211c55c4290abccd366ba0f821b2a5f60660c8b5f6c0ec737"} Nov 03 20:33:07.999997 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:07.999877 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" event={"ID":"fdaa8957-b04b-43f6-8ee4-27212c7f9840","Type":"ContainerStarted","Data":"66e20453cfb55455bac4eec7ef4215d6de4bda197a4fcb8f20358e9d9cfc75b4"} Nov 03 20:33:08.022487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:08.022432 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" podStartSLOduration=2.022413199 podStartE2EDuration="2.022413199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:08.018247601 +0000 UTC m=+2166.393844348" watchObservedRunningTime="2026-11-03 20:33:08.022413199 +0000 UTC m=+2166.398009949" Nov 03 20:33:08.492497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:08.492409 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" podUID="6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:08.492931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:08.492899 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" podUID="fad0ed4c-3ba5-4a1c-a221-668576f5d27e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:09.012500 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:09.012468 4457 generic.go:356] "Generic (PLEG): container finished" podID="435839a4-9123-4b35-83d7-aaa4103ccc28" containerID="92ca6cfa00b6ea30bc0e720c5f9f86629ea6b6cf7fd62585ab791b7652dc2600" exitCode=0 Nov 03 20:33:09.012676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:09.012549 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" event={"ID":"435839a4-9123-4b35-83d7-aaa4103ccc28","Type":"ContainerDied","Data":"92ca6cfa00b6ea30bc0e720c5f9f86629ea6b6cf7fd62585ab791b7652dc2600"} Nov 03 20:33:09.014511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:09.014483 4457 generic.go:356] "Generic (PLEG): container finished" podID="fdaa8957-b04b-43f6-8ee4-27212c7f9840" containerID="8139269e799af25211c55c4290abccd366ba0f821b2a5f60660c8b5f6c0ec737" exitCode=0 Nov 03 20:33:09.014616 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:09.014562 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" event={"ID":"fdaa8957-b04b-43f6-8ee4-27212c7f9840","Type":"ContainerDied","Data":"8139269e799af25211c55c4290abccd366ba0f821b2a5f60660c8b5f6c0ec737"} Nov 03 20:33:10.232028 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.232001 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" Nov 03 20:33:10.273357 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.273331 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" Nov 03 20:33:10.382975 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.382943 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-xks-ca\") pod \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\" (UID: \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\") " Nov 03 20:33:10.383154 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.383033 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/435839a4-9123-4b35-83d7-aaa4103ccc28-kube-api-access-9kwtb\" (UniqueName: \"kubernetes.io/projected/435839a4-9123-4b35-83d7-aaa4103ccc28-kube-api-access-9kwtb\") pod \"435839a4-9123-4b35-83d7-aaa4103ccc28\" (UID: \"435839a4-9123-4b35-83d7-aaa4103ccc28\") " Nov 03 20:33:10.383241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.383165 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-openshift-ca\") pod \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\" (UID: \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\") " Nov 03 20:33:10.383297 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.383261 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-xks-ca\") pod \"435839a4-9123-4b35-83d7-aaa4103ccc28\" (UID: \"435839a4-9123-4b35-83d7-aaa4103ccc28\") " Nov 03 20:33:10.383360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.383320 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdaa8957-b04b-43f6-8ee4-27212c7f9840-kube-api-access-6mc2j\" (UniqueName: \"kubernetes.io/projected/fdaa8957-b04b-43f6-8ee4-27212c7f9840-kube-api-access-6mc2j\") pod \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\" (UID: \"fdaa8957-b04b-43f6-8ee4-27212c7f9840\") " Nov 03 20:33:10.383505 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.383471 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-openshift-ca\") pod \"435839a4-9123-4b35-83d7-aaa4103ccc28\" (UID: \"435839a4-9123-4b35-83d7-aaa4103ccc28\") " Nov 03 20:33:10.383652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.383586 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-openshift-ca" pod "fdaa8957-b04b-43f6-8ee4-27212c7f9840" (UID: "fdaa8957-b04b-43f6-8ee4-27212c7f9840"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:10.384027 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.384001 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-openshift-ca" pod "435839a4-9123-4b35-83d7-aaa4103ccc28" (UID: "435839a4-9123-4b35-83d7-aaa4103ccc28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:10.384206 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.384188 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.384297 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.384213 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.385674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.385649 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/435839a4-9123-4b35-83d7-aaa4103ccc28-kube-api-access-9kwtb" pod "435839a4-9123-4b35-83d7-aaa4103ccc28" (UID: "435839a4-9123-4b35-83d7-aaa4103ccc28"). InnerVolumeSpecName "kube-api-access-9kwtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:10.385674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.385658 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-xks-ca" pod "fdaa8957-b04b-43f6-8ee4-27212c7f9840" (UID: "fdaa8957-b04b-43f6-8ee4-27212c7f9840"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:10.386021 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.386001 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdaa8957-b04b-43f6-8ee4-27212c7f9840-kube-api-access-6mc2j" pod "fdaa8957-b04b-43f6-8ee4-27212c7f9840" (UID: "fdaa8957-b04b-43f6-8ee4-27212c7f9840"). InnerVolumeSpecName "kube-api-access-6mc2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:10.386112 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.386073 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-xks-ca" pod "435839a4-9123-4b35-83d7-aaa4103ccc28" (UID: "435839a4-9123-4b35-83d7-aaa4103ccc28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:10.485750 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.485703 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdaa8957-b04b-43f6-8ee4-27212c7f9840-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.485750 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.485749 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kwtb\" (UniqueName: \"kubernetes.io/projected/435839a4-9123-4b35-83d7-aaa4103ccc28-kube-api-access-9kwtb\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.485976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.485765 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/435839a4-9123-4b35-83d7-aaa4103ccc28-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:10.485976 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:10.485780 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mc2j\" (UniqueName: \"kubernetes.io/projected/fdaa8957-b04b-43f6-8ee4-27212c7f9840-kube-api-access-6mc2j\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.039862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.039830 4457 generic.go:356] "Generic (PLEG): container finished" podID="8d108227-cd53-4d58-9d65-1ff5fd6d1149" containerID="644b7684c182052a788c5daf6815a0d00a650b751249db5a11830f58e16e8db2" exitCode=0 Nov 03 20:33:11.040050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.039911 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" event={"ID":"8d108227-cd53-4d58-9d65-1ff5fd6d1149","Type":"ContainerDied","Data":"644b7684c182052a788c5daf6815a0d00a650b751249db5a11830f58e16e8db2"} Nov 03 20:33:11.041652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.041625 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" event={"ID":"435839a4-9123-4b35-83d7-aaa4103ccc28","Type":"ContainerDied","Data":"56bb7703b9c948ffc1aa236168a8b06186eb0061c0faf536fe5971e4db93c429"} Nov 03 20:33:11.041652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.041646 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl" Nov 03 20:33:11.041829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.041667 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56bb7703b9c948ffc1aa236168a8b06186eb0061c0faf536fe5971e4db93c429" Nov 03 20:33:11.043366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.043349 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" Nov 03 20:33:11.043495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.043417 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k" event={"ID":"fdaa8957-b04b-43f6-8ee4-27212c7f9840","Type":"ContainerDied","Data":"66e20453cfb55455bac4eec7ef4215d6de4bda197a4fcb8f20358e9d9cfc75b4"} Nov 03 20:33:11.043495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.043466 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66e20453cfb55455bac4eec7ef4215d6de4bda197a4fcb8f20358e9d9cfc75b4" Nov 03 20:33:11.637104 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.637078 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:33:11.754194 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.754090 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" containerID="cri-o://be32a44234f74347c436d43d8250ac9fa87eb7764638404d550f9df4a4a8b6d9" gracePeriod=45 Nov 03 20:33:11.811362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.811321 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-kube-api-access-62q5c\" (UniqueName: \"kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-kube-api-access-62q5c\") pod \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " Nov 03 20:33:11.811362 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.811416 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-data\") pod \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " Nov 03 20:33:11.811622 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.811496 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-certs\") pod \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " Nov 03 20:33:11.811622 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.811523 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-credential-socket\") pod \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " Nov 03 20:33:11.811622 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.811573 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-socket\") pod \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " Nov 03 20:33:11.811622 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.811603 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-podinfo\") pod \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " Nov 03 20:33:11.811787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.811640 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-token\" (UniqueName: \"kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-token\") pod \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " Nov 03 20:33:11.811787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.811722 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-envoy\") pod \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " Nov 03 20:33:11.811860 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.811849 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istiod-ca-cert\") pod \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\" (UID: \"8d108227-cd53-4d58-9d65-1ff5fd6d1149\") " Nov 03 20:33:11.812027 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.811996 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-credential-socket" pod "8d108227-cd53-4d58-9d65-1ff5fd6d1149" (UID: "8d108227-cd53-4d58-9d65-1ff5fd6d1149"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.812562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.812537 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-certs" pod "8d108227-cd53-4d58-9d65-1ff5fd6d1149" (UID: "8d108227-cd53-4d58-9d65-1ff5fd6d1149"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.812562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.812520 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-socket" pod "8d108227-cd53-4d58-9d65-1ff5fd6d1149" (UID: "8d108227-cd53-4d58-9d65-1ff5fd6d1149"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.812562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.812555 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-data" pod "8d108227-cd53-4d58-9d65-1ff5fd6d1149" (UID: "8d108227-cd53-4d58-9d65-1ff5fd6d1149"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.812773 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.812717 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istiod-ca-cert" pod "8d108227-cd53-4d58-9d65-1ff5fd6d1149" (UID: "8d108227-cd53-4d58-9d65-1ff5fd6d1149"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:11.812773 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.812749 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.812877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.812776 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.812877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.812793 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.812877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.812809 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.814212 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.814191 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-kube-api-access-62q5c" pod "8d108227-cd53-4d58-9d65-1ff5fd6d1149" (UID: "8d108227-cd53-4d58-9d65-1ff5fd6d1149"). InnerVolumeSpecName "kube-api-access-62q5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:11.814416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.814380 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-token" pod "8d108227-cd53-4d58-9d65-1ff5fd6d1149" (UID: "8d108227-cd53-4d58-9d65-1ff5fd6d1149"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:11.814664 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.814637 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-envoy" pod "8d108227-cd53-4d58-9d65-1ff5fd6d1149" (UID: "8d108227-cd53-4d58-9d65-1ff5fd6d1149"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:11.815136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.815115 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-podinfo" pod "8d108227-cd53-4d58-9d65-1ff5fd6d1149" (UID: "8d108227-cd53-4d58-9d65-1ff5fd6d1149"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:11.822350 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.822296 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" containerID="cri-o://39246feb65cb2de0354f0d0eb827bcb44911089f5050873568127428c3e6cf36" gracePeriod=45 Nov 03 20:33:11.854337 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.854279 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" containerID="cri-o://9df1a760a04b92a6546eadf08c55349bb12dcefbe58719080e97f6a3caebab9a" gracePeriod=45 Nov 03 20:33:11.899314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.899259 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.92:8000/ready\": dial tcp 10.133.0.92:8000: connect: connection refused" Nov 03 20:33:11.913995 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.913957 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.913995 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.913990 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.913995 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.914001 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62q5c\" (UniqueName: \"kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-kube-api-access-62q5c\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.914218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.914012 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:11.914218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:11.914022 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d108227-cd53-4d58-9d65-1ff5fd6d1149-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.072625 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.069685 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" event={"ID":"8d108227-cd53-4d58-9d65-1ff5fd6d1149","Type":"ContainerDied","Data":"01c01b2edb357b087e77e9b9c16c8ac8ba57a96c5d58b54642f9dd4caa372a52"} Nov 03 20:33:12.072625 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.069807 4457 scope.go:122] "RemoveContainer" containerID="644b7684c182052a788c5daf6815a0d00a650b751249db5a11830f58e16e8db2" Nov 03 20:33:12.072625 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.070004 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22" Nov 03 20:33:12.081798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.081605 4457 generic.go:356] "Generic (PLEG): container finished" podID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerID="39246feb65cb2de0354f0d0eb827bcb44911089f5050873568127428c3e6cf36" exitCode=0 Nov 03 20:33:12.081798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.081722 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" event={"ID":"4a9b8725-2131-4b8d-9eae-e2ef60be3627","Type":"ContainerDied","Data":"39246feb65cb2de0354f0d0eb827bcb44911089f5050873568127428c3e6cf36"} Nov 03 20:33:12.084711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.084640 4457 generic.go:356] "Generic (PLEG): container finished" podID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerID="be32a44234f74347c436d43d8250ac9fa87eb7764638404d550f9df4a4a8b6d9" exitCode=0 Nov 03 20:33:12.085131 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.085036 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" event={"ID":"289c7342-c7d8-4281-bcb2-164607ad83ea","Type":"ContainerDied","Data":"be32a44234f74347c436d43d8250ac9fa87eb7764638404d550f9df4a4a8b6d9"} Nov 03 20:33:12.088097 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.088073 4457 generic.go:356] "Generic (PLEG): container finished" podID="d24f9087-0bd1-426c-8182-86be5a40418a" containerID="9df1a760a04b92a6546eadf08c55349bb12dcefbe58719080e97f6a3caebab9a" exitCode=0 Nov 03 20:33:12.088505 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.088485 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" event={"ID":"d24f9087-0bd1-426c-8182-86be5a40418a","Type":"ContainerDied","Data":"9df1a760a04b92a6546eadf08c55349bb12dcefbe58719080e97f6a3caebab9a"} Nov 03 20:33:12.115724 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.115690 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22"] Nov 03 20:33:12.122897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.122866 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7ae0f1-openshift-default-6c665bff95-h4s22"] Nov 03 20:33:12.147467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.147437 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:33:12.240501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.237606 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-home\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-home\") pod \"289c7342-c7d8-4281-bcb2-164607ad83ea\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " Nov 03 20:33:12.240501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.237654 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/289c7342-c7d8-4281-bcb2-164607ad83ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/289c7342-c7d8-4281-bcb2-164607ad83ea-tls-certs\") pod \"289c7342-c7d8-4281-bcb2-164607ad83ea\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " Nov 03 20:33:12.240501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.237766 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-model-cache\") pod \"289c7342-c7d8-4281-bcb2-164607ad83ea\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " Nov 03 20:33:12.240501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.237805 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-dshm\") pod \"289c7342-c7d8-4281-bcb2-164607ad83ea\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " Nov 03 20:33:12.240501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.237894 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289c7342-c7d8-4281-bcb2-164607ad83ea-kube-api-access-vdlxp\" (UniqueName: \"kubernetes.io/projected/289c7342-c7d8-4281-bcb2-164607ad83ea-kube-api-access-vdlxp\") pod \"289c7342-c7d8-4281-bcb2-164607ad83ea\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " Nov 03 20:33:12.240501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.238079 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-tmp-dir\") pod \"289c7342-c7d8-4281-bcb2-164607ad83ea\" (UID: \"289c7342-c7d8-4281-bcb2-164607ad83ea\") " Nov 03 20:33:12.240501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.239149 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-model-cache" pod "289c7342-c7d8-4281-bcb2-164607ad83ea" (UID: "289c7342-c7d8-4281-bcb2-164607ad83ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.240501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.239335 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-home" pod "289c7342-c7d8-4281-bcb2-164607ad83ea" (UID: "289c7342-c7d8-4281-bcb2-164607ad83ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.240501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.239860 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-tmp-dir" pod "289c7342-c7d8-4281-bcb2-164607ad83ea" (UID: "289c7342-c7d8-4281-bcb2-164607ad83ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.244600 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.243030 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289c7342-c7d8-4281-bcb2-164607ad83ea-kube-api-access-vdlxp" pod "289c7342-c7d8-4281-bcb2-164607ad83ea" (UID: "289c7342-c7d8-4281-bcb2-164607ad83ea"). InnerVolumeSpecName "kube-api-access-vdlxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:12.244600 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.243778 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289c7342-c7d8-4281-bcb2-164607ad83ea-tls-certs" pod "289c7342-c7d8-4281-bcb2-164607ad83ea" (UID: "289c7342-c7d8-4281-bcb2-164607ad83ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:12.264802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.259727 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-dshm" pod "289c7342-c7d8-4281-bcb2-164607ad83ea" (UID: "289c7342-c7d8-4281-bcb2-164607ad83ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.308909 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.308560 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:33:12.344087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.340726 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.344087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.340770 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.344087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.340784 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289c7342-c7d8-4281-bcb2-164607ad83ea-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.344087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.340798 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.344087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.340815 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/289c7342-c7d8-4281-bcb2-164607ad83ea-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.344087 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.340829 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdlxp\" (UniqueName: \"kubernetes.io/projected/289c7342-c7d8-4281-bcb2-164607ad83ea-kube-api-access-vdlxp\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.364473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.364021 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:33:12.442060 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442014 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-home\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-home\") pod \"d24f9087-0bd1-426c-8182-86be5a40418a\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " Nov 03 20:33:12.442280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442083 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-model-cache\") pod \"d24f9087-0bd1-426c-8182-86be5a40418a\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " Nov 03 20:33:12.442280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442166 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-tmp-dir\") pod \"d24f9087-0bd1-426c-8182-86be5a40418a\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " Nov 03 20:33:12.442280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442213 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-model-cache\") pod \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " Nov 03 20:33:12.442280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442251 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-dshm\") pod \"d24f9087-0bd1-426c-8182-86be5a40418a\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " Nov 03 20:33:12.442560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442293 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tmp-dir\") pod \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " Nov 03 20:33:12.442560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442331 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-home" pod "d24f9087-0bd1-426c-8182-86be5a40418a" (UID: "d24f9087-0bd1-426c-8182-86be5a40418a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.442560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442359 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tls-certs\") pod \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " Nov 03 20:33:12.442560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442415 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a9b8725-2131-4b8d-9eae-e2ef60be3627-kube-api-access-lrql9\" (UniqueName: \"kubernetes.io/projected/4a9b8725-2131-4b8d-9eae-e2ef60be3627-kube-api-access-lrql9\") pod \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " Nov 03 20:33:12.442560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442472 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d24f9087-0bd1-426c-8182-86be5a40418a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d24f9087-0bd1-426c-8182-86be5a40418a-tls-certs\") pod \"d24f9087-0bd1-426c-8182-86be5a40418a\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " Nov 03 20:33:12.442560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442479 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-tmp-dir" pod "d24f9087-0bd1-426c-8182-86be5a40418a" (UID: "d24f9087-0bd1-426c-8182-86be5a40418a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.442560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442536 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d24f9087-0bd1-426c-8182-86be5a40418a-kube-api-access-j7s55\" (UniqueName: \"kubernetes.io/projected/d24f9087-0bd1-426c-8182-86be5a40418a-kube-api-access-j7s55\") pod \"d24f9087-0bd1-426c-8182-86be5a40418a\" (UID: \"d24f9087-0bd1-426c-8182-86be5a40418a\") " Nov 03 20:33:12.442942 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442578 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-dshm\") pod \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " Nov 03 20:33:12.442942 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442679 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-home\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-home\") pod \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\" (UID: \"4a9b8725-2131-4b8d-9eae-e2ef60be3627\") " Nov 03 20:33:12.442942 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.442888 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tmp-dir" pod "4a9b8725-2131-4b8d-9eae-e2ef60be3627" (UID: "4a9b8725-2131-4b8d-9eae-e2ef60be3627"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.443811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.443130 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-home" pod "4a9b8725-2131-4b8d-9eae-e2ef60be3627" (UID: "4a9b8725-2131-4b8d-9eae-e2ef60be3627"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.443811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.443351 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-model-cache" pod "4a9b8725-2131-4b8d-9eae-e2ef60be3627" (UID: "4a9b8725-2131-4b8d-9eae-e2ef60be3627"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.443811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.443132 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-model-cache" pod "d24f9087-0bd1-426c-8182-86be5a40418a" (UID: "d24f9087-0bd1-426c-8182-86be5a40418a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.443811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.443584 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.443811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.443606 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.443811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.443619 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.443811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.443637 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.443811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.443654 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.443811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.443668 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.446217 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.446185 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d24f9087-0bd1-426c-8182-86be5a40418a-kube-api-access-j7s55" pod "d24f9087-0bd1-426c-8182-86be5a40418a" (UID: "d24f9087-0bd1-426c-8182-86be5a40418a"). InnerVolumeSpecName "kube-api-access-j7s55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:12.446493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.446461 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a9b8725-2131-4b8d-9eae-e2ef60be3627-kube-api-access-lrql9" pod "4a9b8725-2131-4b8d-9eae-e2ef60be3627" (UID: "4a9b8725-2131-4b8d-9eae-e2ef60be3627"). InnerVolumeSpecName "kube-api-access-lrql9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:12.446754 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.446730 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tls-certs" pod "4a9b8725-2131-4b8d-9eae-e2ef60be3627" (UID: "4a9b8725-2131-4b8d-9eae-e2ef60be3627"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:12.447041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.447018 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-dshm" pod "d24f9087-0bd1-426c-8182-86be5a40418a" (UID: "d24f9087-0bd1-426c-8182-86be5a40418a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.447510 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.447485 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-dshm" pod "4a9b8725-2131-4b8d-9eae-e2ef60be3627" (UID: "4a9b8725-2131-4b8d-9eae-e2ef60be3627"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:12.448356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.448327 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d24f9087-0bd1-426c-8182-86be5a40418a-tls-certs" pod "d24f9087-0bd1-426c-8182-86be5a40418a" (UID: "d24f9087-0bd1-426c-8182-86be5a40418a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:12.544837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.544756 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d24f9087-0bd1-426c-8182-86be5a40418a-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.544837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.544787 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a9b8725-2131-4b8d-9eae-e2ef60be3627-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.544837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.544798 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrql9\" (UniqueName: \"kubernetes.io/projected/4a9b8725-2131-4b8d-9eae-e2ef60be3627-kube-api-access-lrql9\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.544837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.544807 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d24f9087-0bd1-426c-8182-86be5a40418a-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.544837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.544816 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7s55\" (UniqueName: \"kubernetes.io/projected/d24f9087-0bd1-426c-8182-86be5a40418a-kube-api-access-j7s55\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.544837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.544824 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a9b8725-2131-4b8d-9eae-e2ef60be3627-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:12.568540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.568497 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" podUID="9deb1243-b43c-4e37-94a9-1eb5e96c21cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:12.568728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.568517 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" podUID="7f934e95-c712-48af-83d4-a7b80d870089" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:12.885973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.885940 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj"] Nov 03 20:33:12.886831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.886811 4457 cpu_manager.go:414] "containerMap: removing container" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" Nov 03 20:33:12.886831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.886833 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" Nov 03 20:33:12.886949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.886841 4457 cpu_manager.go:414] "containerMap: removing container" podUID="435839a4-9123-4b35-83d7-aaa4103ccc28" containerName="revoke-keys" Nov 03 20:33:12.886949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.886847 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="435839a4-9123-4b35-83d7-aaa4103ccc28" containerName="revoke-keys" Nov 03 20:33:12.886949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.886869 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" Nov 03 20:33:12.886949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.886875 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" Nov 03 20:33:12.886949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.886882 4457 cpu_manager.go:414] "containerMap: removing container" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" Nov 03 20:33:12.886949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.886887 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" Nov 03 20:33:12.886949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.886896 4457 cpu_manager.go:414] "containerMap: removing container" podUID="fdaa8957-b04b-43f6-8ee4-27212c7f9840" containerName="revoke-keys" Nov 03 20:33:12.886949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.886901 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdaa8957-b04b-43f6-8ee4-27212c7f9840" containerName="revoke-keys" Nov 03 20:33:12.886949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.886923 4457 cpu_manager.go:414] "containerMap: removing container" podUID="8d108227-cd53-4d58-9d65-1ff5fd6d1149" containerName="istio-proxy" Nov 03 20:33:12.886949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.886927 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d108227-cd53-4d58-9d65-1ff5fd6d1149" containerName="istio-proxy" Nov 03 20:33:12.887576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.887038 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" containerName="main" Nov 03 20:33:12.887576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.887049 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="435839a4-9123-4b35-83d7-aaa4103ccc28" containerName="revoke-keys" Nov 03 20:33:12.887576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.887057 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" containerName="main" Nov 03 20:33:12.887576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.887062 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" Nov 03 20:33:12.887576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.887068 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdaa8957-b04b-43f6-8ee4-27212c7f9840" containerName="revoke-keys" Nov 03 20:33:12.887576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.887079 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d108227-cd53-4d58-9d65-1ff5fd6d1149" containerName="istio-proxy" Nov 03 20:33:12.893027 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.892976 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" Nov 03 20:33:12.898050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.897451 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:33:12.898050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.897696 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:33:12.900296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.900275 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g"] Nov 03 20:33:12.906701 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.906675 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" Nov 03 20:33:12.916326 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.913324 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj"] Nov 03 20:33:12.920769 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.920741 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g"] Nov 03 20:33:12.949097 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.949068 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql6gg\" (UniqueName: \"kubernetes.io/projected/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-kube-api-access-ql6gg\") pod \"maas-api-revoke-sub-9ed006a1-2bhgj\" (UID: \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" Nov 03 20:33:12.949236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.949129 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed006a1-2bhgj\" (UID: \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" Nov 03 20:33:12.949304 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.949287 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed006a1-2bhgj\" (UID: \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" Nov 03 20:33:12.949472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.949451 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e8d85a-pct6g\" (UID: \"13f1debc-c299-423c-91ef-393fb9b8bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" Nov 03 20:33:12.949546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.949496 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jkfx\" (UniqueName: \"kubernetes.io/projected/13f1debc-c299-423c-91ef-393fb9b8bdac-kube-api-access-9jkfx\") pod \"maas-api-revoke-sub-11e8d85a-pct6g\" (UID: \"13f1debc-c299-423c-91ef-393fb9b8bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" Nov 03 20:33:12.949546 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:12.949530 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e8d85a-pct6g\" (UID: \"13f1debc-c299-423c-91ef-393fb9b8bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" Nov 03 20:33:13.050765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.050727 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql6gg\" (UniqueName: \"kubernetes.io/projected/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-kube-api-access-ql6gg\") pod \"maas-api-revoke-sub-9ed006a1-2bhgj\" (UID: \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" Nov 03 20:33:13.050987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.050799 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed006a1-2bhgj\" (UID: \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" Nov 03 20:33:13.050987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.050872 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed006a1-2bhgj\" (UID: \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" Nov 03 20:33:13.050987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.050961 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e8d85a-pct6g\" (UID: \"13f1debc-c299-423c-91ef-393fb9b8bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" Nov 03 20:33:13.050987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.050988 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jkfx\" (UniqueName: \"kubernetes.io/projected/13f1debc-c299-423c-91ef-393fb9b8bdac-kube-api-access-9jkfx\") pod \"maas-api-revoke-sub-11e8d85a-pct6g\" (UID: \"13f1debc-c299-423c-91ef-393fb9b8bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" Nov 03 20:33:13.051218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.051035 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e8d85a-pct6g\" (UID: \"13f1debc-c299-423c-91ef-393fb9b8bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" Nov 03 20:33:13.051706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.051677 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ed006a1-2bhgj\" (UID: \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" Nov 03 20:33:13.051890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.051748 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e8d85a-pct6g\" (UID: \"13f1debc-c299-423c-91ef-393fb9b8bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" Nov 03 20:33:13.053255 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.053232 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ed006a1-2bhgj\" (UID: \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" Nov 03 20:33:13.053448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.053431 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e8d85a-pct6g\" (UID: \"13f1debc-c299-423c-91ef-393fb9b8bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" Nov 03 20:33:13.063553 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.063525 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jkfx\" (UniqueName: \"kubernetes.io/projected/13f1debc-c299-423c-91ef-393fb9b8bdac-kube-api-access-9jkfx\") pod \"maas-api-revoke-sub-11e8d85a-pct6g\" (UID: \"13f1debc-c299-423c-91ef-393fb9b8bdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" Nov 03 20:33:13.063856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.063836 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql6gg\" (UniqueName: \"kubernetes.io/projected/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-kube-api-access-ql6gg\") pod \"maas-api-revoke-sub-9ed006a1-2bhgj\" (UID: \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" Nov 03 20:33:13.100713 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.100673 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" Nov 03 20:33:13.100890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.100680 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" event={"ID":"4a9b8725-2131-4b8d-9eae-e2ef60be3627","Type":"ContainerDied","Data":"5f72cdbce641982a5caf4b3cf7b0973a542b1e5cd367c0d78fd5b848987cb7c6"} Nov 03 20:33:13.100890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.100816 4457 scope.go:122] "RemoveContainer" containerID="39246feb65cb2de0354f0d0eb827bcb44911089f5050873568127428c3e6cf36" Nov 03 20:33:13.102634 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.102603 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" Nov 03 20:33:13.102773 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.102601 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l" event={"ID":"289c7342-c7d8-4281-bcb2-164607ad83ea","Type":"ContainerDied","Data":"9f5d10bb8a9a87cb1a5052629bc650029ebf08941961f1e49a98c3e2300f32e6"} Nov 03 20:33:13.105017 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.104990 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" event={"ID":"d24f9087-0bd1-426c-8182-86be5a40418a","Type":"ContainerDied","Data":"e65500eb59911cc8e1c051fe240d97080e7e38aae34b397c383e0c3c431c8c12"} Nov 03 20:33:13.105163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.105067 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9" Nov 03 20:33:13.115203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.114779 4457 scope.go:122] "RemoveContainer" containerID="be32a44234f74347c436d43d8250ac9fa87eb7764638404d550f9df4a4a8b6d9" Nov 03 20:33:13.126233 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.126215 4457 scope.go:122] "RemoveContainer" containerID="9df1a760a04b92a6546eadf08c55349bb12dcefbe58719080e97f6a3caebab9a" Nov 03 20:33:13.131807 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.131766 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 20:33:13.135564 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.135540 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8"] Nov 03 20:33:13.141265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.141242 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/premium-w2-90ef9c-kserve-5fb48ff7d6-r4rq8"] Nov 03 20:33:13.155089 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.155068 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l"] Nov 03 20:33:13.160714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.160689 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/distinct-2-w2-90ef9c-kserve-6b9cf97d6-v6w4l"] Nov 03 20:33:13.173244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.173220 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9"] Nov 03 20:33:13.180334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.180306 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-90ef9c/simulator-w2-90ef9c-kserve-644c9585b7-t8pz9"] Nov 03 20:33:13.232450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.232423 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" Nov 03 20:33:13.243205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.242376 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" Nov 03 20:33:13.582363 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.582331 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj"] Nov 03 20:33:13.607241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.607207 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g"] Nov 03 20:33:13.607795 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:33:13.607770 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3113d6b_7df2_4bfb_b272_d0a29e1f5e6e.slice/crio-25f4cf23ea05311b2feb79b7711e61a85c14903d21d5e45ac3bce32580bdd8c2 WatchSource:0}: Error finding container 25f4cf23ea05311b2feb79b7711e61a85c14903d21d5e45ac3bce32580bdd8c2: Status 404 returned error can't find the container with id 25f4cf23ea05311b2feb79b7711e61a85c14903d21d5e45ac3bce32580bdd8c2 Nov 03 20:33:13.634574 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:33:13.634544 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f1debc_c299_423c_91ef_393fb9b8bdac.slice/crio-63d43f73c0c1571450ce7e191c01ffeb074bd30385a85469a40dae0ffbeec194 WatchSource:0}: Error finding container 63d43f73c0c1571450ce7e191c01ffeb074bd30385a85469a40dae0ffbeec194: Status 404 returned error can't find the container with id 63d43f73c0c1571450ce7e191c01ffeb074bd30385a85469a40dae0ffbeec194 Nov 03 20:33:13.799186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.796471 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289c7342-c7d8-4281-bcb2-164607ad83ea" path="/var/lib/kubelet/pods/289c7342-c7d8-4281-bcb2-164607ad83ea/volumes" Nov 03 20:33:13.799186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.798926 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a9b8725-2131-4b8d-9eae-e2ef60be3627" path="/var/lib/kubelet/pods/4a9b8725-2131-4b8d-9eae-e2ef60be3627/volumes" Nov 03 20:33:13.804912 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.804872 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d108227-cd53-4d58-9d65-1ff5fd6d1149" path="/var/lib/kubelet/pods/8d108227-cd53-4d58-9d65-1ff5fd6d1149/volumes" Nov 03 20:33:13.813128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.807369 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d24f9087-0bd1-426c-8182-86be5a40418a" path="/var/lib/kubelet/pods/d24f9087-0bd1-426c-8182-86be5a40418a/volumes" Nov 03 20:33:13.892176 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.892152 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" Nov 03 20:33:13.916316 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.916289 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" Nov 03 20:33:13.975233 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.975192 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f934e95-c712-48af-83d4-a7b80d870089-kube-api-access-z2d8j\" (UniqueName: \"kubernetes.io/projected/7f934e95-c712-48af-83d4-a7b80d870089-kube-api-access-z2d8j\") pod \"7f934e95-c712-48af-83d4-a7b80d870089\" (UID: \"7f934e95-c712-48af-83d4-a7b80d870089\") " Nov 03 20:33:13.979460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:13.979424 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f934e95-c712-48af-83d4-a7b80d870089-kube-api-access-z2d8j" pod "7f934e95-c712-48af-83d4-a7b80d870089" (UID: "7f934e95-c712-48af-83d4-a7b80d870089"). InnerVolumeSpecName "kube-api-access-z2d8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:14.076440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.076408 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-plugins-config-volume\") pod \"9deb1243-b43c-4e37-94a9-1eb5e96c21cc\" (UID: \"9deb1243-b43c-4e37-94a9-1eb5e96c21cc\") " Nov 03 20:33:14.076654 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.076638 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-kube-api-access-cch84\" (UniqueName: \"kubernetes.io/projected/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-kube-api-access-cch84\") pod \"9deb1243-b43c-4e37-94a9-1eb5e96c21cc\" (UID: \"9deb1243-b43c-4e37-94a9-1eb5e96c21cc\") " Nov 03 20:33:14.076721 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.076691 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f934e95-c712-48af-83d4-a7b80d870089-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f934e95-c712-48af-83d4-a7b80d870089-plugins-config-volume\") pod \"7f934e95-c712-48af-83d4-a7b80d870089\" (UID: \"7f934e95-c712-48af-83d4-a7b80d870089\") " Nov 03 20:33:14.076797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.076748 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-plugins-config-volume" pod "9deb1243-b43c-4e37-94a9-1eb5e96c21cc" (UID: "9deb1243-b43c-4e37-94a9-1eb5e96c21cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:14.077096 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.077054 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f934e95-c712-48af-83d4-a7b80d870089-plugins-config-volume" pod "7f934e95-c712-48af-83d4-a7b80d870089" (UID: "7f934e95-c712-48af-83d4-a7b80d870089"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:14.077571 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.077549 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f934e95-c712-48af-83d4-a7b80d870089-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:14.077674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.077573 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:14.077674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.077582 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2d8j\" (UniqueName: \"kubernetes.io/projected/7f934e95-c712-48af-83d4-a7b80d870089-kube-api-access-z2d8j\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:14.079359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.079332 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-kube-api-access-cch84" pod "9deb1243-b43c-4e37-94a9-1eb5e96c21cc" (UID: "9deb1243-b43c-4e37-94a9-1eb5e96c21cc"). InnerVolumeSpecName "kube-api-access-cch84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:14.115006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.114972 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" event={"ID":"13f1debc-c299-423c-91ef-393fb9b8bdac","Type":"ContainerStarted","Data":"8e3dfb7945465f895db9f2d1c26159c532fd639cce7995f37f70cde1ba7c8242"} Nov 03 20:33:14.115146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.115025 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" event={"ID":"13f1debc-c299-423c-91ef-393fb9b8bdac","Type":"ContainerStarted","Data":"63d43f73c0c1571450ce7e191c01ffeb074bd30385a85469a40dae0ffbeec194"} Nov 03 20:33:14.116523 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.116496 4457 generic.go:356] "Generic (PLEG): container finished" podID="7f934e95-c712-48af-83d4-a7b80d870089" containerID="769b176777ba6c1539ce2059ff6006c048d1e64dc330e477bb939645bb5ad55f" exitCode=0 Nov 03 20:33:14.116642 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.116532 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" Nov 03 20:33:14.116642 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.116583 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" event={"ID":"7f934e95-c712-48af-83d4-a7b80d870089","Type":"ContainerDied","Data":"769b176777ba6c1539ce2059ff6006c048d1e64dc330e477bb939645bb5ad55f"} Nov 03 20:33:14.116642 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.116633 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff" event={"ID":"7f934e95-c712-48af-83d4-a7b80d870089","Type":"ContainerDied","Data":"3f04d133e980627250d3d0892c7f5c806431c0f094862fe34a36c7aa7236e8ef"} Nov 03 20:33:14.116851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.116664 4457 scope.go:122] "RemoveContainer" containerID="769b176777ba6c1539ce2059ff6006c048d1e64dc330e477bb939645bb5ad55f" Nov 03 20:33:14.120416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.120378 4457 generic.go:356] "Generic (PLEG): container finished" podID="9deb1243-b43c-4e37-94a9-1eb5e96c21cc" containerID="9f3e9cf7c85c34eab1eb05a24c462671d4f5414f348d682ef5595d83e6572195" exitCode=0 Nov 03 20:33:14.120514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.120459 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" Nov 03 20:33:14.120514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.120496 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" event={"ID":"9deb1243-b43c-4e37-94a9-1eb5e96c21cc","Type":"ContainerDied","Data":"9f3e9cf7c85c34eab1eb05a24c462671d4f5414f348d682ef5595d83e6572195"} Nov 03 20:33:14.120641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.120541 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz" event={"ID":"9deb1243-b43c-4e37-94a9-1eb5e96c21cc","Type":"ContainerDied","Data":"4df0289f267fca242e9e1c9ec46757cbc4db0afce006f943f9da3497eb5e70d2"} Nov 03 20:33:14.121829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.121806 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" event={"ID":"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e","Type":"ContainerStarted","Data":"25f4cf23ea05311b2feb79b7711e61a85c14903d21d5e45ac3bce32580bdd8c2"} Nov 03 20:33:14.135980 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.135934 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" podStartSLOduration=2.135917383 podStartE2EDuration="2.135917383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:14.133087386 +0000 UTC m=+2172.508684133" watchObservedRunningTime="2026-11-03 20:33:14.135917383 +0000 UTC m=+2172.511514132" Nov 03 20:33:14.136152 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.135961 4457 scope.go:122] "RemoveContainer" containerID="769b176777ba6c1539ce2059ff6006c048d1e64dc330e477bb939645bb5ad55f" Nov 03 20:33:14.145442 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:14.141338 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"769b176777ba6c1539ce2059ff6006c048d1e64dc330e477bb939645bb5ad55f\": container with ID starting with 769b176777ba6c1539ce2059ff6006c048d1e64dc330e477bb939645bb5ad55f not found: ID does not exist" containerID="769b176777ba6c1539ce2059ff6006c048d1e64dc330e477bb939645bb5ad55f" Nov 03 20:33:14.145442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.141441 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"769b176777ba6c1539ce2059ff6006c048d1e64dc330e477bb939645bb5ad55f"} err="failed to get container status \"769b176777ba6c1539ce2059ff6006c048d1e64dc330e477bb939645bb5ad55f\": rpc error: code = NotFound desc = could not find container \"769b176777ba6c1539ce2059ff6006c048d1e64dc330e477bb939645bb5ad55f\": container with ID starting with 769b176777ba6c1539ce2059ff6006c048d1e64dc330e477bb939645bb5ad55f not found: ID does not exist" Nov 03 20:33:14.145442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.141472 4457 scope.go:122] "RemoveContainer" containerID="9f3e9cf7c85c34eab1eb05a24c462671d4f5414f348d682ef5595d83e6572195" Nov 03 20:33:14.186340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.179585 4457 scope.go:122] "RemoveContainer" containerID="9f3e9cf7c85c34eab1eb05a24c462671d4f5414f348d682ef5595d83e6572195" Nov 03 20:33:14.186340 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:14.180949 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f3e9cf7c85c34eab1eb05a24c462671d4f5414f348d682ef5595d83e6572195\": container with ID starting with 9f3e9cf7c85c34eab1eb05a24c462671d4f5414f348d682ef5595d83e6572195 not found: ID does not exist" containerID="9f3e9cf7c85c34eab1eb05a24c462671d4f5414f348d682ef5595d83e6572195" Nov 03 20:33:14.186340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.180992 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f3e9cf7c85c34eab1eb05a24c462671d4f5414f348d682ef5595d83e6572195"} err="failed to get container status \"9f3e9cf7c85c34eab1eb05a24c462671d4f5414f348d682ef5595d83e6572195\": rpc error: code = NotFound desc = could not find container \"9f3e9cf7c85c34eab1eb05a24c462671d4f5414f348d682ef5595d83e6572195\": container with ID starting with 9f3e9cf7c85c34eab1eb05a24c462671d4f5414f348d682ef5595d83e6572195 not found: ID does not exist" Nov 03 20:33:14.195785 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.195730 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cch84\" (UniqueName: \"kubernetes.io/projected/9deb1243-b43c-4e37-94a9-1eb5e96c21cc-kube-api-access-cch84\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:14.203444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.203418 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz"] Nov 03 20:33:14.213804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.213775 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6cf8ae-d566b589d-ftwcz"] Nov 03 20:33:14.222828 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.222798 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff"] Nov 03 20:33:14.228365 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:14.228342 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6cf8ae-6dd98c4687-n52ff"] Nov 03 20:33:15.134078 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:15.134040 4457 generic.go:356] "Generic (PLEG): container finished" podID="a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e" containerID="fda88f3734455df6dad4aa7842d0a0dd9052845abec08483440e3a1122b61a84" exitCode=0 Nov 03 20:33:15.134583 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:15.134129 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" event={"ID":"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e","Type":"ContainerDied","Data":"fda88f3734455df6dad4aa7842d0a0dd9052845abec08483440e3a1122b61a84"} Nov 03 20:33:15.135930 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:15.135904 4457 generic.go:356] "Generic (PLEG): container finished" podID="13f1debc-c299-423c-91ef-393fb9b8bdac" containerID="8e3dfb7945465f895db9f2d1c26159c532fd639cce7995f37f70cde1ba7c8242" exitCode=0 Nov 03 20:33:15.136073 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:15.135981 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" event={"ID":"13f1debc-c299-423c-91ef-393fb9b8bdac","Type":"ContainerDied","Data":"8e3dfb7945465f895db9f2d1c26159c532fd639cce7995f37f70cde1ba7c8242"} Nov 03 20:33:15.762571 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:15.762535 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f934e95-c712-48af-83d4-a7b80d870089" path="/var/lib/kubelet/pods/7f934e95-c712-48af-83d4-a7b80d870089/volumes" Nov 03 20:33:15.763184 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:15.763161 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9deb1243-b43c-4e37-94a9-1eb5e96c21cc" path="/var/lib/kubelet/pods/9deb1243-b43c-4e37-94a9-1eb5e96c21cc/volumes" Nov 03 20:33:16.375140 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.375111 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" Nov 03 20:33:16.380578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.380465 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" Nov 03 20:33:16.438009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.433135 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f1debc-c299-423c-91ef-393fb9b8bdac-kube-api-access-9jkfx\" (UniqueName: \"kubernetes.io/projected/13f1debc-c299-423c-91ef-393fb9b8bdac-kube-api-access-9jkfx\") pod \"13f1debc-c299-423c-91ef-393fb9b8bdac\" (UID: \"13f1debc-c299-423c-91ef-393fb9b8bdac\") " Nov 03 20:33:16.438009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.433203 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-openshift-ca\") pod \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\" (UID: \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\") " Nov 03 20:33:16.438009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.433256 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-xks-ca\") pod \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\" (UID: \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\") " Nov 03 20:33:16.438009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.433302 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-openshift-ca\") pod \"13f1debc-c299-423c-91ef-393fb9b8bdac\" (UID: \"13f1debc-c299-423c-91ef-393fb9b8bdac\") " Nov 03 20:33:16.438009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.433498 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-xks-ca\") pod \"13f1debc-c299-423c-91ef-393fb9b8bdac\" (UID: \"13f1debc-c299-423c-91ef-393fb9b8bdac\") " Nov 03 20:33:16.438009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.433554 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-kube-api-access-ql6gg\" (UniqueName: \"kubernetes.io/projected/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-kube-api-access-ql6gg\") pod \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\" (UID: \"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e\") " Nov 03 20:33:16.461241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.460972 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-openshift-ca" pod "a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e" (UID: "a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:16.466076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.466030 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f1debc-c299-423c-91ef-393fb9b8bdac-kube-api-access-9jkfx" pod "13f1debc-c299-423c-91ef-393fb9b8bdac" (UID: "13f1debc-c299-423c-91ef-393fb9b8bdac"). InnerVolumeSpecName "kube-api-access-9jkfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:16.467725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.467693 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-openshift-ca" pod "13f1debc-c299-423c-91ef-393fb9b8bdac" (UID: "13f1debc-c299-423c-91ef-393fb9b8bdac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:16.468768 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.468706 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-kube-api-access-ql6gg" pod "a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e" (UID: "a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e"). InnerVolumeSpecName "kube-api-access-ql6gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:16.468768 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.468746 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-xks-ca" pod "a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e" (UID: "a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:16.473890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.473853 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-xks-ca" pod "13f1debc-c299-423c-91ef-393fb9b8bdac" (UID: "13f1debc-c299-423c-91ef-393fb9b8bdac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:16.535352 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.535314 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jkfx\" (UniqueName: \"kubernetes.io/projected/13f1debc-c299-423c-91ef-393fb9b8bdac-kube-api-access-9jkfx\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:16.535352 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.535348 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:16.535597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.535361 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:16.535597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.535370 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:16.535597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.535382 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f1debc-c299-423c-91ef-393fb9b8bdac-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:16.535597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.535409 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql6gg\" (UniqueName: \"kubernetes.io/projected/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e-kube-api-access-ql6gg\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:16.604776 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.604739 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk"] Nov 03 20:33:16.617185 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:16.617149 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6"] Nov 03 20:33:17.156506 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.156468 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" Nov 03 20:33:17.156506 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.156495 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj" event={"ID":"a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e","Type":"ContainerDied","Data":"25f4cf23ea05311b2feb79b7711e61a85c14903d21d5e45ac3bce32580bdd8c2"} Nov 03 20:33:17.156788 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.156550 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25f4cf23ea05311b2feb79b7711e61a85c14903d21d5e45ac3bce32580bdd8c2" Nov 03 20:33:17.162438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.158338 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" Nov 03 20:33:17.162438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.158368 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g" event={"ID":"13f1debc-c299-423c-91ef-393fb9b8bdac","Type":"ContainerDied","Data":"63d43f73c0c1571450ce7e191c01ffeb074bd30385a85469a40dae0ffbeec194"} Nov 03 20:33:17.162438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.158427 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63d43f73c0c1571450ce7e191c01ffeb074bd30385a85469a40dae0ffbeec194" Nov 03 20:33:17.351731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.351683 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4"] Nov 03 20:33:17.721464 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.716608 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" Nov 03 20:33:17.869406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.850286 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-kube-api-access-l5lqv\" (UniqueName: \"kubernetes.io/projected/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-kube-api-access-l5lqv\") pod \"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7\" (UID: \"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7\") " Nov 03 20:33:17.869406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.850447 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-plugins-config-volume\") pod \"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7\" (UID: \"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7\") " Nov 03 20:33:17.869406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.851487 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-plugins-config-volume" pod "6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7" (UID: "6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:17.910494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.909714 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-kube-api-access-l5lqv" pod "6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7" (UID: "6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7"). InnerVolumeSpecName "kube-api-access-l5lqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:17.943918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.943894 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" Nov 03 20:33:17.952279 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.952252 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-plugins-config-volume\") pod \"fad0ed4c-3ba5-4a1c-a221-668576f5d27e\" (UID: \"fad0ed4c-3ba5-4a1c-a221-668576f5d27e\") " Nov 03 20:33:17.952473 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.952458 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-kube-api-access-qp9dm\" (UniqueName: \"kubernetes.io/projected/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-kube-api-access-qp9dm\") pod \"fad0ed4c-3ba5-4a1c-a221-668576f5d27e\" (UID: \"fad0ed4c-3ba5-4a1c-a221-668576f5d27e\") " Nov 03 20:33:17.952715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.952689 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-plugins-config-volume" pod "fad0ed4c-3ba5-4a1c-a221-668576f5d27e" (UID: "fad0ed4c-3ba5-4a1c-a221-668576f5d27e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:17.952980 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.952954 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5lqv\" (UniqueName: \"kubernetes.io/projected/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-kube-api-access-l5lqv\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:17.952980 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.952976 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:17.953091 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.952992 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:17.954535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:17.954514 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-kube-api-access-qp9dm" pod "fad0ed4c-3ba5-4a1c-a221-668576f5d27e" (UID: "fad0ed4c-3ba5-4a1c-a221-668576f5d27e"). InnerVolumeSpecName "kube-api-access-qp9dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:18.054835 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.054800 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp9dm\" (UniqueName: \"kubernetes.io/projected/fad0ed4c-3ba5-4a1c-a221-668576f5d27e-kube-api-access-qp9dm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:18.170276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.170235 4457 generic.go:356] "Generic (PLEG): container finished" podID="fad0ed4c-3ba5-4a1c-a221-668576f5d27e" containerID="91d322d9a065425bc4e4284396cc9c8878aa58ac59c9e4ce80bd6a27ed5503d5" exitCode=0 Nov 03 20:33:18.170495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.170285 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" Nov 03 20:33:18.170495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.170335 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" event={"ID":"fad0ed4c-3ba5-4a1c-a221-668576f5d27e","Type":"ContainerDied","Data":"91d322d9a065425bc4e4284396cc9c8878aa58ac59c9e4ce80bd6a27ed5503d5"} Nov 03 20:33:18.170495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.170413 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9" event={"ID":"fad0ed4c-3ba5-4a1c-a221-668576f5d27e","Type":"ContainerDied","Data":"a017406fd4f9f8997f1d34c7bea19bc3146b6c32622f61e10d9241fb1379aec8"} Nov 03 20:33:18.170495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.170448 4457 scope.go:122] "RemoveContainer" containerID="91d322d9a065425bc4e4284396cc9c8878aa58ac59c9e4ce80bd6a27ed5503d5" Nov 03 20:33:18.172659 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.172633 4457 generic.go:356] "Generic (PLEG): container finished" podID="6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7" containerID="4d051ef325f777b6f42b652fc23c61da1dadc49e790c42debe1211bed9e1700f" exitCode=0 Nov 03 20:33:18.172795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.172676 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" Nov 03 20:33:18.172795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.172729 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" event={"ID":"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7","Type":"ContainerDied","Data":"4d051ef325f777b6f42b652fc23c61da1dadc49e790c42debe1211bed9e1700f"} Nov 03 20:33:18.172908 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.172800 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5" event={"ID":"6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7","Type":"ContainerDied","Data":"1dda7eb4726066b7b2db15a8f7089e7ffaf5d59dfc53c018384477eb5d2817fb"} Nov 03 20:33:18.193635 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.192518 4457 scope.go:122] "RemoveContainer" containerID="91d322d9a065425bc4e4284396cc9c8878aa58ac59c9e4ce80bd6a27ed5503d5" Nov 03 20:33:18.193752 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:18.193708 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d322d9a065425bc4e4284396cc9c8878aa58ac59c9e4ce80bd6a27ed5503d5\": container with ID starting with 91d322d9a065425bc4e4284396cc9c8878aa58ac59c9e4ce80bd6a27ed5503d5 not found: ID does not exist" containerID="91d322d9a065425bc4e4284396cc9c8878aa58ac59c9e4ce80bd6a27ed5503d5" Nov 03 20:33:18.193835 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.193782 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d322d9a065425bc4e4284396cc9c8878aa58ac59c9e4ce80bd6a27ed5503d5"} err="failed to get container status \"91d322d9a065425bc4e4284396cc9c8878aa58ac59c9e4ce80bd6a27ed5503d5\": rpc error: code = NotFound desc = could not find container \"91d322d9a065425bc4e4284396cc9c8878aa58ac59c9e4ce80bd6a27ed5503d5\": container with ID starting with 91d322d9a065425bc4e4284396cc9c8878aa58ac59c9e4ce80bd6a27ed5503d5 not found: ID does not exist" Nov 03 20:33:18.193835 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.193807 4457 scope.go:122] "RemoveContainer" containerID="4d051ef325f777b6f42b652fc23c61da1dadc49e790c42debe1211bed9e1700f" Nov 03 20:33:18.206084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.206058 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9"] Nov 03 20:33:18.207455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.207434 4457 scope.go:122] "RemoveContainer" containerID="4d051ef325f777b6f42b652fc23c61da1dadc49e790c42debe1211bed9e1700f" Nov 03 20:33:18.207770 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:18.207751 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d051ef325f777b6f42b652fc23c61da1dadc49e790c42debe1211bed9e1700f\": container with ID starting with 4d051ef325f777b6f42b652fc23c61da1dadc49e790c42debe1211bed9e1700f not found: ID does not exist" containerID="4d051ef325f777b6f42b652fc23c61da1dadc49e790c42debe1211bed9e1700f" Nov 03 20:33:18.207835 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.207780 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d051ef325f777b6f42b652fc23c61da1dadc49e790c42debe1211bed9e1700f"} err="failed to get container status \"4d051ef325f777b6f42b652fc23c61da1dadc49e790c42debe1211bed9e1700f\": rpc error: code = NotFound desc = could not find container \"4d051ef325f777b6f42b652fc23c61da1dadc49e790c42debe1211bed9e1700f\": container with ID starting with 4d051ef325f777b6f42b652fc23c61da1dadc49e790c42debe1211bed9e1700f not found: ID does not exist" Nov 03 20:33:18.211053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.211032 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7ae0f1-ff9dd9647-wtjq9"] Nov 03 20:33:18.222973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.222950 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5"] Nov 03 20:33:18.230203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:18.230182 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7ae0f1-595c47d7c6-rqjf5"] Nov 03 20:33:19.761624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:19.761590 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7" path="/var/lib/kubelet/pods/6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7/volumes" Nov 03 20:33:19.765356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:19.762129 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad0ed4c-3ba5-4a1c-a221-668576f5d27e" path="/var/lib/kubelet/pods/fad0ed4c-3ba5-4a1c-a221-668576f5d27e/volumes" Nov 03 20:33:20.669408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:20.665828 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k"] Nov 03 20:33:20.798410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:20.794533 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d0898b-8847k"] Nov 03 20:33:21.605263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:21.605210 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" podUID="2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5" containerName="payload-processing" containerID="cri-o://60b1925b30e045fe0a5a8713c8f9335b7b491e1b1593fcad57414712cc39ac4e" gracePeriod=25 Nov 03 20:33:21.618136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:21.618093 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" podUID="47128d77-8e6f-4e60-9299-585cfe98b268" containerName="payload-pre-processing" containerID="cri-o://ab4a7020c483e3e0f5099e44cc3a94718cc1094aac4a7ac1e2a58c918523a224" gracePeriod=25 Nov 03 20:33:21.776715 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:21.776678 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdaa8957-b04b-43f6-8ee4-27212c7f9840" path="/var/lib/kubelet/pods/fdaa8957-b04b-43f6-8ee4-27212c7f9840/volumes" Nov 03 20:33:22.352729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:22.352667 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" podUID="7b3116a9-eaba-42f2-a977-082a8228165b" containerName="payload-pre-processing" containerID="cri-o://88a63af6d65677da2f4fbc49d31a5461cc114e77fc55cfef874e652f59be8635" gracePeriod=25 Nov 03 20:33:23.208711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.208664 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb"] Nov 03 20:33:23.209456 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209438 4457 cpu_manager.go:414] "containerMap: removing container" podUID="9deb1243-b43c-4e37-94a9-1eb5e96c21cc" containerName="payload-processing" Nov 03 20:33:23.209527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209458 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9deb1243-b43c-4e37-94a9-1eb5e96c21cc" containerName="payload-processing" Nov 03 20:33:23.209527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209482 4457 cpu_manager.go:414] "containerMap: removing container" podUID="fad0ed4c-3ba5-4a1c-a221-668576f5d27e" containerName="payload-pre-processing" Nov 03 20:33:23.209527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209489 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad0ed4c-3ba5-4a1c-a221-668576f5d27e" containerName="payload-pre-processing" Nov 03 20:33:23.209527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209509 4457 cpu_manager.go:414] "containerMap: removing container" podUID="6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7" containerName="payload-processing" Nov 03 20:33:23.209527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209514 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7" containerName="payload-processing" Nov 03 20:33:23.209527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209523 4457 cpu_manager.go:414] "containerMap: removing container" podUID="13f1debc-c299-423c-91ef-393fb9b8bdac" containerName="revoke-keys" Nov 03 20:33:23.209527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209528 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f1debc-c299-423c-91ef-393fb9b8bdac" containerName="revoke-keys" Nov 03 20:33:23.209755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209545 4457 cpu_manager.go:414] "containerMap: removing container" podUID="a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e" containerName="revoke-keys" Nov 03 20:33:23.209755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209550 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e" containerName="revoke-keys" Nov 03 20:33:23.209755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209557 4457 cpu_manager.go:414] "containerMap: removing container" podUID="7f934e95-c712-48af-83d4-a7b80d870089" containerName="payload-pre-processing" Nov 03 20:33:23.209755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209561 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f934e95-c712-48af-83d4-a7b80d870089" containerName="payload-pre-processing" Nov 03 20:33:23.209755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209675 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f1debc-c299-423c-91ef-393fb9b8bdac" containerName="revoke-keys" Nov 03 20:33:23.209755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209693 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e" containerName="revoke-keys" Nov 03 20:33:23.209755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209708 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f934e95-c712-48af-83d4-a7b80d870089" containerName="payload-pre-processing" Nov 03 20:33:23.209755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209721 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="9deb1243-b43c-4e37-94a9-1eb5e96c21cc" containerName="payload-processing" Nov 03 20:33:23.209755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209732 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad0ed4c-3ba5-4a1c-a221-668576f5d27e" containerName="payload-pre-processing" Nov 03 20:33:23.209755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.209744 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec607bd-4c2a-4925-8f3a-85aa2a56b7e7" containerName="payload-processing" Nov 03 20:33:23.215215 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.214854 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" Nov 03 20:33:23.222862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.222825 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb"] Nov 03 20:33:23.285172 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.285129 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:33:23.285172 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.285165 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:33:23.367799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.367753 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gspm\" (UniqueName: \"kubernetes.io/projected/68c7127d-b076-426c-b797-dc8046d4f074-kube-api-access-7gspm\") pod \"maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb\" (UID: \"68c7127d-b076-426c-b797-dc8046d4f074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" Nov 03 20:33:23.368276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.367920 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb\" (UID: \"68c7127d-b076-426c-b797-dc8046d4f074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" Nov 03 20:33:23.368276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.368176 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb\" (UID: \"68c7127d-b076-426c-b797-dc8046d4f074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" Nov 03 20:33:23.470150 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.470054 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb\" (UID: \"68c7127d-b076-426c-b797-dc8046d4f074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" Nov 03 20:33:23.470342 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.470233 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb\" (UID: \"68c7127d-b076-426c-b797-dc8046d4f074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" Nov 03 20:33:23.470342 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.470283 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gspm\" (UniqueName: \"kubernetes.io/projected/68c7127d-b076-426c-b797-dc8046d4f074-kube-api-access-7gspm\") pod \"maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb\" (UID: \"68c7127d-b076-426c-b797-dc8046d4f074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" Nov 03 20:33:23.470881 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.470855 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb\" (UID: \"68c7127d-b076-426c-b797-dc8046d4f074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" Nov 03 20:33:23.472740 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.472720 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb\" (UID: \"68c7127d-b076-426c-b797-dc8046d4f074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" Nov 03 20:33:23.482857 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.482829 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gspm\" (UniqueName: \"kubernetes.io/projected/68c7127d-b076-426c-b797-dc8046d4f074-kube-api-access-7gspm\") pod \"maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb\" (UID: \"68c7127d-b076-426c-b797-dc8046d4f074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" Nov 03 20:33:23.532694 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.532661 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" Nov 03 20:33:23.839477 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:33:23.839438 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c7127d_b076_426c_b797_dc8046d4f074.slice/crio-2e6280d3cd9401467c9c720e34521ae2190716dfef2fba6ebedb2993ebfbe72b WatchSource:0}: Error finding container 2e6280d3cd9401467c9c720e34521ae2190716dfef2fba6ebedb2993ebfbe72b: Status 404 returned error can't find the container with id 2e6280d3cd9401467c9c720e34521ae2190716dfef2fba6ebedb2993ebfbe72b Nov 03 20:33:23.839629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:23.839526 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb"] Nov 03 20:33:24.239833 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:24.239794 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" event={"ID":"68c7127d-b076-426c-b797-dc8046d4f074","Type":"ContainerStarted","Data":"247a7b4603bd4d458657cb5746ece7a017eb3cba7b67089ce9f7177b7b394096"} Nov 03 20:33:24.240028 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:24.239854 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" event={"ID":"68c7127d-b076-426c-b797-dc8046d4f074","Type":"ContainerStarted","Data":"2e6280d3cd9401467c9c720e34521ae2190716dfef2fba6ebedb2993ebfbe72b"} Nov 03 20:33:24.270343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:24.270270 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" podStartSLOduration=1.2702505849999999 podStartE2EDuration="1.270250585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:24.265562772 +0000 UTC m=+2182.641159518" watchObservedRunningTime="2026-11-03 20:33:24.270250585 +0000 UTC m=+2182.645847333" Nov 03 20:33:25.252846 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:25.252814 4457 generic.go:356] "Generic (PLEG): container finished" podID="68c7127d-b076-426c-b797-dc8046d4f074" containerID="247a7b4603bd4d458657cb5746ece7a017eb3cba7b67089ce9f7177b7b394096" exitCode=0 Nov 03 20:33:25.253257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:25.252898 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" event={"ID":"68c7127d-b076-426c-b797-dc8046d4f074","Type":"ContainerDied","Data":"247a7b4603bd4d458657cb5746ece7a017eb3cba7b67089ce9f7177b7b394096"} Nov 03 20:33:26.432619 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:26.432594 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" Nov 03 20:33:26.525245 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:26.525099 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-openshift-ca\") pod \"68c7127d-b076-426c-b797-dc8046d4f074\" (UID: \"68c7127d-b076-426c-b797-dc8046d4f074\") " Nov 03 20:33:26.525534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:26.525289 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-xks-ca\") pod \"68c7127d-b076-426c-b797-dc8046d4f074\" (UID: \"68c7127d-b076-426c-b797-dc8046d4f074\") " Nov 03 20:33:26.525534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:26.525443 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68c7127d-b076-426c-b797-dc8046d4f074-kube-api-access-7gspm\" (UniqueName: \"kubernetes.io/projected/68c7127d-b076-426c-b797-dc8046d4f074-kube-api-access-7gspm\") pod \"68c7127d-b076-426c-b797-dc8046d4f074\" (UID: \"68c7127d-b076-426c-b797-dc8046d4f074\") " Nov 03 20:33:26.525681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:26.525661 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-openshift-ca" pod "68c7127d-b076-426c-b797-dc8046d4f074" (UID: "68c7127d-b076-426c-b797-dc8046d4f074"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:26.526183 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:26.526136 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:26.529186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:26.529159 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-xks-ca" pod "68c7127d-b076-426c-b797-dc8046d4f074" (UID: "68c7127d-b076-426c-b797-dc8046d4f074"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:26.555431 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:26.537569 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c7127d-b076-426c-b797-dc8046d4f074-kube-api-access-7gspm" pod "68c7127d-b076-426c-b797-dc8046d4f074" (UID: "68c7127d-b076-426c-b797-dc8046d4f074"). InnerVolumeSpecName "kube-api-access-7gspm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:26.637423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:26.629841 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68c7127d-b076-426c-b797-dc8046d4f074-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:26.637423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:26.629887 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gspm\" (UniqueName: \"kubernetes.io/projected/68c7127d-b076-426c-b797-dc8046d4f074-kube-api-access-7gspm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:27.277099 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:27.277066 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" Nov 03 20:33:27.277099 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:27.277081 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb" event={"ID":"68c7127d-b076-426c-b797-dc8046d4f074","Type":"ContainerDied","Data":"2e6280d3cd9401467c9c720e34521ae2190716dfef2fba6ebedb2993ebfbe72b"} Nov 03 20:33:27.277413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:27.277130 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e6280d3cd9401467c9c720e34521ae2190716dfef2fba6ebedb2993ebfbe72b" Nov 03 20:33:29.722419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.713419 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" podUID="2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:29.722419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.714659 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" podUID="47128d77-8e6f-4e60-9299-585cfe98b268" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:29.722419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.717224 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt"] Nov 03 20:33:29.722419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.718335 4457 cpu_manager.go:414] "containerMap: removing container" podUID="68c7127d-b076-426c-b797-dc8046d4f074" containerName="revoke-keys" Nov 03 20:33:29.722419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.718357 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68c7127d-b076-426c-b797-dc8046d4f074" containerName="revoke-keys" Nov 03 20:33:29.722419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.718539 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="68c7127d-b076-426c-b797-dc8046d4f074" containerName="revoke-keys" Nov 03 20:33:29.730424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.727554 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" Nov 03 20:33:29.741435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.735554 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt"] Nov 03 20:33:29.801080 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.801047 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:33:29.801254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.801082 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:33:29.887979 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.887940 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gprtm\" (UniqueName: \"kubernetes.io/projected/7e03f447-8195-4c9f-a9b5-6dd64724624a-kube-api-access-gprtm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt\" (UID: \"7e03f447-8195-4c9f-a9b5-6dd64724624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" Nov 03 20:33:29.888192 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.888118 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt\" (UID: \"7e03f447-8195-4c9f-a9b5-6dd64724624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" Nov 03 20:33:29.888270 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.888193 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt\" (UID: \"7e03f447-8195-4c9f-a9b5-6dd64724624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" Nov 03 20:33:29.990206 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.990117 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt\" (UID: \"7e03f447-8195-4c9f-a9b5-6dd64724624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" Nov 03 20:33:29.990206 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.990180 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt\" (UID: \"7e03f447-8195-4c9f-a9b5-6dd64724624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" Nov 03 20:33:29.990908 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.990605 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gprtm\" (UniqueName: \"kubernetes.io/projected/7e03f447-8195-4c9f-a9b5-6dd64724624a-kube-api-access-gprtm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt\" (UID: \"7e03f447-8195-4c9f-a9b5-6dd64724624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" Nov 03 20:33:29.991156 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.991124 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt\" (UID: \"7e03f447-8195-4c9f-a9b5-6dd64724624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" Nov 03 20:33:29.993085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:29.993026 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt\" (UID: \"7e03f447-8195-4c9f-a9b5-6dd64724624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" Nov 03 20:33:30.003539 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:30.003512 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gprtm\" (UniqueName: \"kubernetes.io/projected/7e03f447-8195-4c9f-a9b5-6dd64724624a-kube-api-access-gprtm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt\" (UID: \"7e03f447-8195-4c9f-a9b5-6dd64724624a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" Nov 03 20:33:30.060742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:30.060705 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" Nov 03 20:33:30.352067 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:33:30.352028 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e03f447_8195_4c9f_a9b5_6dd64724624a.slice/crio-e2ef3c3afef79c0f1a1e1ffc50c5068eff12bc3094af29e896efedcf3efbbd12 WatchSource:0}: Error finding container e2ef3c3afef79c0f1a1e1ffc50c5068eff12bc3094af29e896efedcf3efbbd12: Status 404 returned error can't find the container with id e2ef3c3afef79c0f1a1e1ffc50c5068eff12bc3094af29e896efedcf3efbbd12 Nov 03 20:33:30.352745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:30.352603 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt"] Nov 03 20:33:31.319250 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:31.319217 4457 generic.go:356] "Generic (PLEG): container finished" podID="7e03f447-8195-4c9f-a9b5-6dd64724624a" containerID="ec45db5e71e9c0b5f0126ac1fd22bd2a6d7ef1f761e4032eeee8a26a163b3114" exitCode=0 Nov 03 20:33:31.319664 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:31.319258 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" event={"ID":"7e03f447-8195-4c9f-a9b5-6dd64724624a","Type":"ContainerDied","Data":"ec45db5e71e9c0b5f0126ac1fd22bd2a6d7ef1f761e4032eeee8a26a163b3114"} Nov 03 20:33:31.319664 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:31.319326 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" event={"ID":"7e03f447-8195-4c9f-a9b5-6dd64724624a","Type":"ContainerStarted","Data":"e2ef3c3afef79c0f1a1e1ffc50c5068eff12bc3094af29e896efedcf3efbbd12"} Nov 03 20:33:31.328518 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:31.328489 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" podUID="7b3116a9-eaba-42f2-a977-082a8228165b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:32.645057 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:32.645018 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl"] Nov 03 20:33:32.650586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:32.650558 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-5ca474-pfnnl"] Nov 03 20:33:32.808409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:32.805147 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" Nov 03 20:33:32.939804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:32.939772 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e03f447-8195-4c9f-a9b5-6dd64724624a-kube-api-access-gprtm\" (UniqueName: \"kubernetes.io/projected/7e03f447-8195-4c9f-a9b5-6dd64724624a-kube-api-access-gprtm\") pod \"7e03f447-8195-4c9f-a9b5-6dd64724624a\" (UID: \"7e03f447-8195-4c9f-a9b5-6dd64724624a\") " Nov 03 20:33:32.939996 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:32.939981 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-xks-ca\") pod \"7e03f447-8195-4c9f-a9b5-6dd64724624a\" (UID: \"7e03f447-8195-4c9f-a9b5-6dd64724624a\") " Nov 03 20:33:32.940071 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:32.940039 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-openshift-ca\") pod \"7e03f447-8195-4c9f-a9b5-6dd64724624a\" (UID: \"7e03f447-8195-4c9f-a9b5-6dd64724624a\") " Nov 03 20:33:32.940749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:32.940383 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-openshift-ca" pod "7e03f447-8195-4c9f-a9b5-6dd64724624a" (UID: "7e03f447-8195-4c9f-a9b5-6dd64724624a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:32.940897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:32.940821 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:32.942251 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:32.942227 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-xks-ca" pod "7e03f447-8195-4c9f-a9b5-6dd64724624a" (UID: "7e03f447-8195-4c9f-a9b5-6dd64724624a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:32.942251 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:32.942239 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e03f447-8195-4c9f-a9b5-6dd64724624a-kube-api-access-gprtm" pod "7e03f447-8195-4c9f-a9b5-6dd64724624a" (UID: "7e03f447-8195-4c9f-a9b5-6dd64724624a"). InnerVolumeSpecName "kube-api-access-gprtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:33.042163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:33.042080 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e03f447-8195-4c9f-a9b5-6dd64724624a-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:33.042163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:33.042119 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gprtm\" (UniqueName: \"kubernetes.io/projected/7e03f447-8195-4c9f-a9b5-6dd64724624a-kube-api-access-gprtm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:33.355683 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:33.355649 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq"] Nov 03 20:33:33.426343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:33.426315 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" Nov 03 20:33:33.428857 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:33.428567 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt" event={"ID":"7e03f447-8195-4c9f-a9b5-6dd64724624a","Type":"ContainerDied","Data":"e2ef3c3afef79c0f1a1e1ffc50c5068eff12bc3094af29e896efedcf3efbbd12"} Nov 03 20:33:33.429107 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:33.429078 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2ef3c3afef79c0f1a1e1ffc50c5068eff12bc3094af29e896efedcf3efbbd12" Nov 03 20:33:33.821409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:33.811521 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="435839a4-9123-4b35-83d7-aaa4103ccc28" path="/var/lib/kubelet/pods/435839a4-9123-4b35-83d7-aaa4103ccc28/volumes" Nov 03 20:33:34.201475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:34.187073 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd"] Nov 03 20:33:34.259413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:34.254960 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-90ef9c-wwgvd"] Nov 03 20:33:35.762490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:35.762448 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b991442-a874-43f1-bf91-001d877344eb" path="/var/lib/kubelet/pods/6b991442-a874-43f1-bf91-001d877344eb/volumes" Nov 03 20:33:36.150808 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:36.150770 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:33:38.356133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:38.356088 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" podUID="7e57f420-40a0-4f1b-8b58-70f0be591ec8" containerName="payload-pre-processing" containerID="cri-o://d20c1f13b2783e6999c0019d611fed0d580684674f82bf2c36372d162d9bcd01" gracePeriod=25 Nov 03 20:33:39.705118 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:39.705076 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" podUID="47128d77-8e6f-4e60-9299-585cfe98b268" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:39.705629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:39.705076 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" podUID="2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:39.853979 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:39.853944 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh"] Nov 03 20:33:40.469574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:40.469525 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t"] Nov 03 20:33:40.469816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:40.469786 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" podUID="38d44a0d-26ce-4622-b30f-f2c6e013828b" containerName="istio-proxy" containerID="cri-o://2f09ffd899f4c501dc266e1e206b2ad12db4c4fd2f12a4ce84aea172aaa09966" gracePeriod=30 Nov 03 20:33:40.477263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:40.477233 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk"] Nov 03 20:33:40.477563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:40.477536 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" podUID="06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" containerName="istio-proxy" containerID="cri-o://abf13da326034ef676cc116886c83ad68f0b07e56860a51b9d1f504c018119db" gracePeriod=30 Nov 03 20:33:40.900436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:40.900380 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" podUID="7e57f420-40a0-4f1b-8b58-70f0be591ec8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:41.328010 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:41.327970 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" podUID="7b3116a9-eaba-42f2-a977-082a8228165b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:41.938178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:41.938152 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" Nov 03 20:33:42.036311 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.035917 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" Nov 03 20:33:42.048039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.048008 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-plugins-config-volume\") pod \"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5\" (UID: \"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5\") " Nov 03 20:33:42.048211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.048168 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-kube-api-access-zhkp5\" (UniqueName: \"kubernetes.io/projected/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-kube-api-access-zhkp5\") pod \"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5\" (UID: \"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5\") " Nov 03 20:33:42.048500 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.048470 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-plugins-config-volume" pod "2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5" (UID: "2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:42.048871 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.048852 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:42.050413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.050379 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-kube-api-access-zhkp5" pod "2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5" (UID: "2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5"). InnerVolumeSpecName "kube-api-access-zhkp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:42.150047 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.149972 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47128d77-8e6f-4e60-9299-585cfe98b268-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47128d77-8e6f-4e60-9299-585cfe98b268-plugins-config-volume\") pod \"47128d77-8e6f-4e60-9299-585cfe98b268\" (UID: \"47128d77-8e6f-4e60-9299-585cfe98b268\") " Nov 03 20:33:42.150207 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.150080 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47128d77-8e6f-4e60-9299-585cfe98b268-kube-api-access-c24gb\" (UniqueName: \"kubernetes.io/projected/47128d77-8e6f-4e60-9299-585cfe98b268-kube-api-access-c24gb\") pod \"47128d77-8e6f-4e60-9299-585cfe98b268\" (UID: \"47128d77-8e6f-4e60-9299-585cfe98b268\") " Nov 03 20:33:42.150333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.150310 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47128d77-8e6f-4e60-9299-585cfe98b268-plugins-config-volume" pod "47128d77-8e6f-4e60-9299-585cfe98b268" (UID: "47128d77-8e6f-4e60-9299-585cfe98b268"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:42.150641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.150625 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47128d77-8e6f-4e60-9299-585cfe98b268-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:42.150693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.150645 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhkp5\" (UniqueName: \"kubernetes.io/projected/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5-kube-api-access-zhkp5\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:42.152131 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.152100 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47128d77-8e6f-4e60-9299-585cfe98b268-kube-api-access-c24gb" pod "47128d77-8e6f-4e60-9299-585cfe98b268" (UID: "47128d77-8e6f-4e60-9299-585cfe98b268"). InnerVolumeSpecName "kube-api-access-c24gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:42.252302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.252264 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c24gb\" (UniqueName: \"kubernetes.io/projected/47128d77-8e6f-4e60-9299-585cfe98b268-kube-api-access-c24gb\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:42.331864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.331826 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26"] Nov 03 20:33:42.332127 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.332097 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" podUID="94b5e918-acce-4fd9-ad4a-c37d87965667" containerName="istio-proxy" containerID="cri-o://7c4d688bf3e753541e2e7efe52d0579b9c3086d36814b5657b4f4339b86184df" gracePeriod=30 Nov 03 20:33:42.351228 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.351187 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z"] Nov 03 20:33:42.351779 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.351709 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" podUID="a9cd7625-f8fb-42aa-a39a-5ea33b4b8831" containerName="maas-api" containerID="cri-o://74a64a3c1a06a597a4750e476d28352ea1e4b32e2b94717ea73edb5b0cb01bab" gracePeriod=30 Nov 03 20:33:42.561983 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.559109 4457 generic.go:356] "Generic (PLEG): container finished" podID="7b3116a9-eaba-42f2-a977-082a8228165b" containerID="88a63af6d65677da2f4fbc49d31a5461cc114e77fc55cfef874e652f59be8635" exitCode=0 Nov 03 20:33:42.561983 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.559230 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" event={"ID":"7b3116a9-eaba-42f2-a977-082a8228165b","Type":"ContainerDied","Data":"88a63af6d65677da2f4fbc49d31a5461cc114e77fc55cfef874e652f59be8635"} Nov 03 20:33:42.561983 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.561283 4457 generic.go:356] "Generic (PLEG): container finished" podID="47128d77-8e6f-4e60-9299-585cfe98b268" containerID="ab4a7020c483e3e0f5099e44cc3a94718cc1094aac4a7ac1e2a58c918523a224" exitCode=0 Nov 03 20:33:42.561983 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.561506 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" event={"ID":"47128d77-8e6f-4e60-9299-585cfe98b268","Type":"ContainerDied","Data":"ab4a7020c483e3e0f5099e44cc3a94718cc1094aac4a7ac1e2a58c918523a224"} Nov 03 20:33:42.561983 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.561686 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" event={"ID":"47128d77-8e6f-4e60-9299-585cfe98b268","Type":"ContainerDied","Data":"1621e400114baea2ec538e4dbceceadfc5d21ff466065dd3d1f51f57697abbfa"} Nov 03 20:33:42.561983 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.561714 4457 scope.go:122] "RemoveContainer" containerID="ab4a7020c483e3e0f5099e44cc3a94718cc1094aac4a7ac1e2a58c918523a224" Nov 03 20:33:42.561983 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.561928 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6" Nov 03 20:33:42.823349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.818320 4457 generic.go:356] "Generic (PLEG): container finished" podID="2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5" containerID="60b1925b30e045fe0a5a8713c8f9335b7b491e1b1593fcad57414712cc39ac4e" exitCode=0 Nov 03 20:33:42.823349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.818740 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" event={"ID":"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5","Type":"ContainerDied","Data":"60b1925b30e045fe0a5a8713c8f9335b7b491e1b1593fcad57414712cc39ac4e"} Nov 03 20:33:42.823349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.818793 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" event={"ID":"2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5","Type":"ContainerDied","Data":"4997a3a871b65f08e498da5cde97c96c42e309053ca299f35452e68b82d69d5f"} Nov 03 20:33:42.823349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.818878 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk" Nov 03 20:33:42.838252 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.838097 4457 generic.go:356] "Generic (PLEG): container finished" podID="a9cd7625-f8fb-42aa-a39a-5ea33b4b8831" containerID="74a64a3c1a06a597a4750e476d28352ea1e4b32e2b94717ea73edb5b0cb01bab" exitCode=0 Nov 03 20:33:42.838252 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.838181 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" event={"ID":"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831","Type":"ContainerDied","Data":"74a64a3c1a06a597a4750e476d28352ea1e4b32e2b94717ea73edb5b0cb01bab"} Nov 03 20:33:42.868722 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.868659 4457 scope.go:122] "RemoveContainer" containerID="ab4a7020c483e3e0f5099e44cc3a94718cc1094aac4a7ac1e2a58c918523a224" Nov 03 20:33:42.869526 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:42.869123 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4a7020c483e3e0f5099e44cc3a94718cc1094aac4a7ac1e2a58c918523a224\": container with ID starting with ab4a7020c483e3e0f5099e44cc3a94718cc1094aac4a7ac1e2a58c918523a224 not found: ID does not exist" containerID="ab4a7020c483e3e0f5099e44cc3a94718cc1094aac4a7ac1e2a58c918523a224" Nov 03 20:33:42.869526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.869163 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4a7020c483e3e0f5099e44cc3a94718cc1094aac4a7ac1e2a58c918523a224"} err="failed to get container status \"ab4a7020c483e3e0f5099e44cc3a94718cc1094aac4a7ac1e2a58c918523a224\": rpc error: code = NotFound desc = could not find container \"ab4a7020c483e3e0f5099e44cc3a94718cc1094aac4a7ac1e2a58c918523a224\": container with ID starting with ab4a7020c483e3e0f5099e44cc3a94718cc1094aac4a7ac1e2a58c918523a224 not found: ID does not exist" Nov 03 20:33:42.869526 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.869185 4457 scope.go:122] "RemoveContainer" containerID="60b1925b30e045fe0a5a8713c8f9335b7b491e1b1593fcad57414712cc39ac4e" Nov 03 20:33:42.879510 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.879458 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6"] Nov 03 20:33:42.904411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.901357 4457 scope.go:122] "RemoveContainer" containerID="60b1925b30e045fe0a5a8713c8f9335b7b491e1b1593fcad57414712cc39ac4e" Nov 03 20:33:42.905291 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:42.905111 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b1925b30e045fe0a5a8713c8f9335b7b491e1b1593fcad57414712cc39ac4e\": container with ID starting with 60b1925b30e045fe0a5a8713c8f9335b7b491e1b1593fcad57414712cc39ac4e not found: ID does not exist" containerID="60b1925b30e045fe0a5a8713c8f9335b7b491e1b1593fcad57414712cc39ac4e" Nov 03 20:33:42.905291 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.905160 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b1925b30e045fe0a5a8713c8f9335b7b491e1b1593fcad57414712cc39ac4e"} err="failed to get container status \"60b1925b30e045fe0a5a8713c8f9335b7b491e1b1593fcad57414712cc39ac4e\": rpc error: code = NotFound desc = could not find container \"60b1925b30e045fe0a5a8713c8f9335b7b491e1b1593fcad57414712cc39ac4e\": container with ID starting with 60b1925b30e045fe0a5a8713c8f9335b7b491e1b1593fcad57414712cc39ac4e not found: ID does not exist" Nov 03 20:33:42.910273 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.909924 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-d0898b-bd968fc9f-z6vb6"] Nov 03 20:33:42.917057 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.917001 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk"] Nov 03 20:33:42.932108 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.931628 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d0898b-6bd8dc6756-5k8qk"] Nov 03 20:33:42.946864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.941752 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt"] Nov 03 20:33:42.955474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:42.955439 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3ef4b9-5vcqt"] Nov 03 20:33:43.009838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.009799 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" Nov 03 20:33:43.080654 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.080578 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:33:43.115284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.115254 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b3116a9-eaba-42f2-a977-082a8228165b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b3116a9-eaba-42f2-a977-082a8228165b-plugins-config-volume\") pod \"7b3116a9-eaba-42f2-a977-082a8228165b\" (UID: \"7b3116a9-eaba-42f2-a977-082a8228165b\") " Nov 03 20:33:43.115847 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.115825 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3116a9-eaba-42f2-a977-082a8228165b-kube-api-access-ztqmz\" (UniqueName: \"kubernetes.io/projected/7b3116a9-eaba-42f2-a977-082a8228165b-kube-api-access-ztqmz\") pod \"7b3116a9-eaba-42f2-a977-082a8228165b\" (UID: \"7b3116a9-eaba-42f2-a977-082a8228165b\") " Nov 03 20:33:43.116247 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.115846 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b3116a9-eaba-42f2-a977-082a8228165b-plugins-config-volume" pod "7b3116a9-eaba-42f2-a977-082a8228165b" (UID: "7b3116a9-eaba-42f2-a977-082a8228165b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:43.116621 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.116601 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b3116a9-eaba-42f2-a977-082a8228165b-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:43.118630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.118332 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3116a9-eaba-42f2-a977-082a8228165b-kube-api-access-ztqmz" pod "7b3116a9-eaba-42f2-a977-082a8228165b" (UID: "7b3116a9-eaba-42f2-a977-082a8228165b"). InnerVolumeSpecName "kube-api-access-ztqmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:43.218671 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.218618 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-metrics-certs\") pod \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " Nov 03 20:33:43.219025 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.219000 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-maas-api-tls\") pod \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " Nov 03 20:33:43.219281 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.219261 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-openshift-service-ca\") pod \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " Nov 03 20:33:43.219532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.219513 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-kube-api-access-976g7\" (UniqueName: \"kubernetes.io/projected/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-kube-api-access-976g7\") pod \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\" (UID: \"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831\") " Nov 03 20:33:43.220352 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.220313 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztqmz\" (UniqueName: \"kubernetes.io/projected/7b3116a9-eaba-42f2-a977-082a8228165b-kube-api-access-ztqmz\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:43.221506 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.221455 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-openshift-service-ca" pod "a9cd7625-f8fb-42aa-a39a-5ea33b4b8831" (UID: "a9cd7625-f8fb-42aa-a39a-5ea33b4b8831"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:43.221743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.221715 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-metrics-certs" pod "a9cd7625-f8fb-42aa-a39a-5ea33b4b8831" (UID: "a9cd7625-f8fb-42aa-a39a-5ea33b4b8831"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:43.223092 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.223068 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-kube-api-access-976g7" pod "a9cd7625-f8fb-42aa-a39a-5ea33b4b8831" (UID: "a9cd7625-f8fb-42aa-a39a-5ea33b4b8831"). InnerVolumeSpecName "kube-api-access-976g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:43.223218 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.223161 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-maas-api-tls" pod "a9cd7625-f8fb-42aa-a39a-5ea33b4b8831" (UID: "a9cd7625-f8fb-42aa-a39a-5ea33b4b8831"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:43.322288 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.322253 4457 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-metrics-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:43.322288 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.322288 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-maas-api-tls\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:43.322288 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.322304 4457 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-openshift-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:43.322670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.322318 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-976g7\" (UniqueName: \"kubernetes.io/projected/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831-kube-api-access-976g7\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:43.770857 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.770816 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5" path="/var/lib/kubelet/pods/2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5/volumes" Nov 03 20:33:43.772354 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.772328 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47128d77-8e6f-4e60-9299-585cfe98b268" path="/var/lib/kubelet/pods/47128d77-8e6f-4e60-9299-585cfe98b268/volumes" Nov 03 20:33:43.773486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.773468 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e03f447-8195-4c9f-a9b5-6dd64724624a" path="/var/lib/kubelet/pods/7e03f447-8195-4c9f-a9b5-6dd64724624a/volumes" Nov 03 20:33:43.850247 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.850214 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" Nov 03 20:33:43.850482 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.850210 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4" event={"ID":"7b3116a9-eaba-42f2-a977-082a8228165b","Type":"ContainerDied","Data":"188c53db6c064306333de9987b43ad6ce2c613d3432134cd2674642a3ef619a4"} Nov 03 20:33:43.850482 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.850444 4457 scope.go:122] "RemoveContainer" containerID="88a63af6d65677da2f4fbc49d31a5461cc114e77fc55cfef874e652f59be8635" Nov 03 20:33:43.855587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.855562 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" Nov 03 20:33:43.855587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.855571 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z" event={"ID":"a9cd7625-f8fb-42aa-a39a-5ea33b4b8831","Type":"ContainerDied","Data":"36a773ec3987408d1b02d00b0327b1fe455b2bbec247df2add40da736d7a6bae"} Nov 03 20:33:43.900689 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.892356 4457 scope.go:122] "RemoveContainer" containerID="74a64a3c1a06a597a4750e476d28352ea1e4b32e2b94717ea73edb5b0cb01bab" Nov 03 20:33:43.967846 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:43.936365 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4"] Nov 03 20:33:44.073407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:44.069259 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-5ca474-7fb88744cf-2kzd4"] Nov 03 20:33:44.147572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:44.133475 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z"] Nov 03 20:33:44.147572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:44.144441 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-3ef4b9-67f64bc86d-fx92z"] Nov 03 20:33:44.217638 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:44.217574 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb"] Nov 03 20:33:44.225452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:44.225423 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b0adda-wctjb"] Nov 03 20:33:44.855365 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:44.855269 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" podUID="61c89239-0110-425d-9aff-2652c9b8dfd7" containerName="payload-pre-processing" containerID="cri-o://60739149f7f2a5dae5934897651e352c66ebb2628949eeb43852f81f29093c0b" gracePeriod=25 Nov 03 20:33:45.154188 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.154161 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" Nov 03 20:33:45.294436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.281836 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c89239-0110-425d-9aff-2652c9b8dfd7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c89239-0110-425d-9aff-2652c9b8dfd7-plugins-config-volume\") pod \"61c89239-0110-425d-9aff-2652c9b8dfd7\" (UID: \"61c89239-0110-425d-9aff-2652c9b8dfd7\") " Nov 03 20:33:45.294436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.288585 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c89239-0110-425d-9aff-2652c9b8dfd7-kube-api-access-cqqd6\" (UniqueName: \"kubernetes.io/projected/61c89239-0110-425d-9aff-2652c9b8dfd7-kube-api-access-cqqd6\") pod \"61c89239-0110-425d-9aff-2652c9b8dfd7\" (UID: \"61c89239-0110-425d-9aff-2652c9b8dfd7\") " Nov 03 20:33:45.294436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.284072 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c89239-0110-425d-9aff-2652c9b8dfd7-plugins-config-volume" pod "61c89239-0110-425d-9aff-2652c9b8dfd7" (UID: "61c89239-0110-425d-9aff-2652c9b8dfd7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:45.302328 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.302283 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c89239-0110-425d-9aff-2652c9b8dfd7-kube-api-access-cqqd6" pod "61c89239-0110-425d-9aff-2652c9b8dfd7" (UID: "61c89239-0110-425d-9aff-2652c9b8dfd7"). InnerVolumeSpecName "kube-api-access-cqqd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.390353 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.390274 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61c89239-0110-425d-9aff-2652c9b8dfd7-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.390353 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.390304 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqqd6\" (UniqueName: \"kubernetes.io/projected/61c89239-0110-425d-9aff-2652c9b8dfd7-kube-api-access-cqqd6\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.749586 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.749548 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:33:45.783635 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.783596 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68c7127d-b076-426c-b797-dc8046d4f074" path="/var/lib/kubelet/pods/68c7127d-b076-426c-b797-dc8046d4f074/volumes" Nov 03 20:33:45.784297 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.784273 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b3116a9-eaba-42f2-a977-082a8228165b" path="/var/lib/kubelet/pods/7b3116a9-eaba-42f2-a977-082a8228165b/volumes" Nov 03 20:33:45.784962 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.784940 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9cd7625-f8fb-42aa-a39a-5ea33b4b8831" path="/var/lib/kubelet/pods/a9cd7625-f8fb-42aa-a39a-5ea33b4b8831/volumes" Nov 03 20:33:45.803582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.802909 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-envoy\") pod \"38d44a0d-26ce-4622-b30f-f2c6e013828b\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " Nov 03 20:33:45.804027 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.804001 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-kube-api-access-n9v65\" (UniqueName: \"kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-kube-api-access-n9v65\") pod \"38d44a0d-26ce-4622-b30f-f2c6e013828b\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " Nov 03 20:33:45.804128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.804084 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-credential-socket\") pod \"38d44a0d-26ce-4622-b30f-f2c6e013828b\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " Nov 03 20:33:45.804128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.804112 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-token\" (UniqueName: \"kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-token\") pod \"38d44a0d-26ce-4622-b30f-f2c6e013828b\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " Nov 03 20:33:45.804242 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.804154 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-podinfo\") pod \"38d44a0d-26ce-4622-b30f-f2c6e013828b\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " Nov 03 20:33:45.804301 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.804252 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-socket\") pod \"38d44a0d-26ce-4622-b30f-f2c6e013828b\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " Nov 03 20:33:45.804812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.804321 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d44a0d-26ce-4622-b30f-f2c6e013828b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d44a0d-26ce-4622-b30f-f2c6e013828b-istiod-ca-cert\") pod \"38d44a0d-26ce-4622-b30f-f2c6e013828b\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " Nov 03 20:33:45.805264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.805237 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-credential-socket" pod "38d44a0d-26ce-4622-b30f-f2c6e013828b" (UID: "38d44a0d-26ce-4622-b30f-f2c6e013828b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.805343 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.805312 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-socket" pod "38d44a0d-26ce-4622-b30f-f2c6e013828b" (UID: "38d44a0d-26ce-4622-b30f-f2c6e013828b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.805709 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.805687 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d44a0d-26ce-4622-b30f-f2c6e013828b-istiod-ca-cert" pod "38d44a0d-26ce-4622-b30f-f2c6e013828b" (UID: "38d44a0d-26ce-4622-b30f-f2c6e013828b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:45.806081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.806059 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-certs\") pod \"38d44a0d-26ce-4622-b30f-f2c6e013828b\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " Nov 03 20:33:45.806434 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.806378 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-certs" pod "38d44a0d-26ce-4622-b30f-f2c6e013828b" (UID: "38d44a0d-26ce-4622-b30f-f2c6e013828b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.806758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.806734 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-data" pod "38d44a0d-26ce-4622-b30f-f2c6e013828b" (UID: "38d44a0d-26ce-4622-b30f-f2c6e013828b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.806864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.806770 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-data\") pod \"38d44a0d-26ce-4622-b30f-f2c6e013828b\" (UID: \"38d44a0d-26ce-4622-b30f-f2c6e013828b\") " Nov 03 20:33:45.807301 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.807279 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38d44a0d-26ce-4622-b30f-f2c6e013828b-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.807467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.807305 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.807467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.807320 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.807467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.807334 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.807467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.807349 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.808501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.808480 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-podinfo" pod "38d44a0d-26ce-4622-b30f-f2c6e013828b" (UID: "38d44a0d-26ce-4622-b30f-f2c6e013828b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:45.812699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.812668 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-envoy" pod "38d44a0d-26ce-4622-b30f-f2c6e013828b" (UID: "38d44a0d-26ce-4622-b30f-f2c6e013828b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.812919 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.812899 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-token" pod "38d44a0d-26ce-4622-b30f-f2c6e013828b" (UID: "38d44a0d-26ce-4622-b30f-f2c6e013828b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.813972 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.813942 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-kube-api-access-n9v65" pod "38d44a0d-26ce-4622-b30f-f2c6e013828b" (UID: "38d44a0d-26ce-4622-b30f-f2c6e013828b"). InnerVolumeSpecName "kube-api-access-n9v65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.839345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.839314 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:33:45.911287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.911190 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istiod-ca-cert\") pod \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " Nov 03 20:33:45.911492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.911465 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-envoy\") pod \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " Nov 03 20:33:45.911569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.911521 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-data\") pod \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " Nov 03 20:33:45.912237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.912202 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-token\" (UniqueName: \"kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-token\") pod \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " Nov 03 20:33:45.912371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.912250 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istiod-ca-cert" pod "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" (UID: "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:45.912371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.912284 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-credential-socket\") pod \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " Nov 03 20:33:45.912519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.912451 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-certs\") pod \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " Nov 03 20:33:45.912578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.912524 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-kube-api-access-qlcq6\" (UniqueName: \"kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-kube-api-access-qlcq6\") pod \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " Nov 03 20:33:45.912753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.912734 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-socket\") pod \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " Nov 03 20:33:45.912821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.912807 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-podinfo\") pod \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\" (UID: \"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18\") " Nov 03 20:33:45.914714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.914687 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-token" pod "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" (UID: "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.915661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.915634 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-data" pod "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" (UID: "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.915969 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.915947 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-certs" pod "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" (UID: "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.916212 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.916185 4457 generic.go:356] "Generic (PLEG): container finished" podID="61c89239-0110-425d-9aff-2652c9b8dfd7" containerID="60739149f7f2a5dae5934897651e352c66ebb2628949eeb43852f81f29093c0b" exitCode=0 Nov 03 20:33:45.916320 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.916297 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.916358 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.916330 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.916358 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.916335 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-credential-socket" pod "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" (UID: "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.916358 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.916346 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9v65\" (UniqueName: \"kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-kube-api-access-n9v65\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.916492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.916368 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.916492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.916426 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38d44a0d-26ce-4622-b30f-f2c6e013828b-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.916492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.916440 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:45.916588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.916480 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" event={"ID":"61c89239-0110-425d-9aff-2652c9b8dfd7","Type":"ContainerDied","Data":"60739149f7f2a5dae5934897651e352c66ebb2628949eeb43852f81f29093c0b"} Nov 03 20:33:45.916623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.916582 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" event={"ID":"61c89239-0110-425d-9aff-2652c9b8dfd7","Type":"ContainerDied","Data":"577fd7d1b16d7495d786cde00558545906c339f0771e0aca38b8bf3432bcc779"} Nov 03 20:33:45.916778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.916757 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-socket" pod "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" (UID: "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.916858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.916840 4457 scope.go:122] "RemoveContainer" containerID="60739149f7f2a5dae5934897651e352c66ebb2628949eeb43852f81f29093c0b" Nov 03 20:33:45.917358 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.916982 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh" Nov 03 20:33:45.919604 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.919342 4457 generic.go:356] "Generic (PLEG): container finished" podID="38d44a0d-26ce-4622-b30f-f2c6e013828b" containerID="2f09ffd899f4c501dc266e1e206b2ad12db4c4fd2f12a4ce84aea172aaa09966" exitCode=0 Nov 03 20:33:45.919604 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.919512 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-kube-api-access-qlcq6" pod "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" (UID: "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18"). InnerVolumeSpecName "kube-api-access-qlcq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:45.919778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.919633 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" Nov 03 20:33:45.919841 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.919702 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" event={"ID":"38d44a0d-26ce-4622-b30f-f2c6e013828b","Type":"ContainerDied","Data":"2f09ffd899f4c501dc266e1e206b2ad12db4c4fd2f12a4ce84aea172aaa09966"} Nov 03 20:33:45.919841 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.919823 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t" event={"ID":"38d44a0d-26ce-4622-b30f-f2c6e013828b","Type":"ContainerDied","Data":"5a92b0312e0807f1e62a735883b7819090697a0a25f33c58deb2453969ffdf9b"} Nov 03 20:33:45.923623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.921587 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-podinfo" pod "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" (UID: "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:45.923623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.922784 4457 generic.go:356] "Generic (PLEG): container finished" podID="06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" containerID="abf13da326034ef676cc116886c83ad68f0b07e56860a51b9d1f504c018119db" exitCode=0 Nov 03 20:33:45.923623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.922840 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" event={"ID":"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18","Type":"ContainerDied","Data":"abf13da326034ef676cc116886c83ad68f0b07e56860a51b9d1f504c018119db"} Nov 03 20:33:45.923623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.922889 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" event={"ID":"06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18","Type":"ContainerDied","Data":"de38d172fd1bf797cdb1334aca0969facf9add070cee581a42355432697a7c10"} Nov 03 20:33:45.923623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.922994 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk" Nov 03 20:33:45.929909 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.929864 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-envoy" pod "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" (UID: "06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:45.959663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.959629 4457 scope.go:122] "RemoveContainer" containerID="60739149f7f2a5dae5934897651e352c66ebb2628949eeb43852f81f29093c0b" Nov 03 20:33:45.960019 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:45.959999 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60739149f7f2a5dae5934897651e352c66ebb2628949eeb43852f81f29093c0b\": container with ID starting with 60739149f7f2a5dae5934897651e352c66ebb2628949eeb43852f81f29093c0b not found: ID does not exist" containerID="60739149f7f2a5dae5934897651e352c66ebb2628949eeb43852f81f29093c0b" Nov 03 20:33:45.960085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.960028 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60739149f7f2a5dae5934897651e352c66ebb2628949eeb43852f81f29093c0b"} err="failed to get container status \"60739149f7f2a5dae5934897651e352c66ebb2628949eeb43852f81f29093c0b\": rpc error: code = NotFound desc = could not find container \"60739149f7f2a5dae5934897651e352c66ebb2628949eeb43852f81f29093c0b\": container with ID starting with 60739149f7f2a5dae5934897651e352c66ebb2628949eeb43852f81f29093c0b not found: ID does not exist" Nov 03 20:33:45.960085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.960047 4457 scope.go:122] "RemoveContainer" containerID="2f09ffd899f4c501dc266e1e206b2ad12db4c4fd2f12a4ce84aea172aaa09966" Nov 03 20:33:45.973731 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.973704 4457 scope.go:122] "RemoveContainer" containerID="2f09ffd899f4c501dc266e1e206b2ad12db4c4fd2f12a4ce84aea172aaa09966" Nov 03 20:33:45.974106 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:45.974084 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f09ffd899f4c501dc266e1e206b2ad12db4c4fd2f12a4ce84aea172aaa09966\": container with ID starting with 2f09ffd899f4c501dc266e1e206b2ad12db4c4fd2f12a4ce84aea172aaa09966 not found: ID does not exist" containerID="2f09ffd899f4c501dc266e1e206b2ad12db4c4fd2f12a4ce84aea172aaa09966" Nov 03 20:33:45.974199 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.974112 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f09ffd899f4c501dc266e1e206b2ad12db4c4fd2f12a4ce84aea172aaa09966"} err="failed to get container status \"2f09ffd899f4c501dc266e1e206b2ad12db4c4fd2f12a4ce84aea172aaa09966\": rpc error: code = NotFound desc = could not find container \"2f09ffd899f4c501dc266e1e206b2ad12db4c4fd2f12a4ce84aea172aaa09966\": container with ID starting with 2f09ffd899f4c501dc266e1e206b2ad12db4c4fd2f12a4ce84aea172aaa09966 not found: ID does not exist" Nov 03 20:33:45.974199 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.974129 4457 scope.go:122] "RemoveContainer" containerID="abf13da326034ef676cc116886c83ad68f0b07e56860a51b9d1f504c018119db" Nov 03 20:33:45.979742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.979715 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t"] Nov 03 20:33:45.984621 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.984593 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-jgd2t"] Nov 03 20:33:45.987479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.987457 4457 scope.go:122] "RemoveContainer" containerID="abf13da326034ef676cc116886c83ad68f0b07e56860a51b9d1f504c018119db" Nov 03 20:33:45.987811 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:45.987789 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf13da326034ef676cc116886c83ad68f0b07e56860a51b9d1f504c018119db\": container with ID starting with abf13da326034ef676cc116886c83ad68f0b07e56860a51b9d1f504c018119db not found: ID does not exist" containerID="abf13da326034ef676cc116886c83ad68f0b07e56860a51b9d1f504c018119db" Nov 03 20:33:45.987885 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.987820 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf13da326034ef676cc116886c83ad68f0b07e56860a51b9d1f504c018119db"} err="failed to get container status \"abf13da326034ef676cc116886c83ad68f0b07e56860a51b9d1f504c018119db\": rpc error: code = NotFound desc = could not find container \"abf13da326034ef676cc116886c83ad68f0b07e56860a51b9d1f504c018119db\": container with ID starting with abf13da326034ef676cc116886c83ad68f0b07e56860a51b9d1f504c018119db not found: ID does not exist" Nov 03 20:33:45.996060 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:45.996028 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh"] Nov 03 20:33:46.000710 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:46.000677 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3ef4b9-7bd8cd75cf-vx7dh"] Nov 03 20:33:46.017794 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:46.017758 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.017794 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:46.017794 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.018001 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:46.017810 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.018001 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:46.017823 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.018001 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:46.017834 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlcq6\" (UniqueName: \"kubernetes.io/projected/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-kube-api-access-qlcq6\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.018001 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:46.017843 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.018001 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:46.017851 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:46.251708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:46.251672 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk"] Nov 03 20:33:46.256685 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:46.256657 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5ca474-openshift-default-596f897f7-wfkbk"] Nov 03 20:33:47.547752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.547724 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:33:47.651069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.650987 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-token\" (UniqueName: \"kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-token\") pod \"94b5e918-acce-4fd9-ad4a-c37d87965667\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " Nov 03 20:33:47.651069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.651053 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b5e918-acce-4fd9-ad4a-c37d87965667-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b5e918-acce-4fd9-ad4a-c37d87965667-istiod-ca-cert\") pod \"94b5e918-acce-4fd9-ad4a-c37d87965667\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " Nov 03 20:33:47.651458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.651107 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-kube-api-access-vp4zs\" (UniqueName: \"kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-kube-api-access-vp4zs\") pod \"94b5e918-acce-4fd9-ad4a-c37d87965667\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " Nov 03 20:33:47.651458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.651189 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-podinfo\") pod \"94b5e918-acce-4fd9-ad4a-c37d87965667\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " Nov 03 20:33:47.651458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.651226 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-data\") pod \"94b5e918-acce-4fd9-ad4a-c37d87965667\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " Nov 03 20:33:47.651458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.651253 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-socket\") pod \"94b5e918-acce-4fd9-ad4a-c37d87965667\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " Nov 03 20:33:47.651458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.651339 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-certs\") pod \"94b5e918-acce-4fd9-ad4a-c37d87965667\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " Nov 03 20:33:47.651458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.651452 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-envoy\") pod \"94b5e918-acce-4fd9-ad4a-c37d87965667\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " Nov 03 20:33:47.651776 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.651489 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-credential-socket\") pod \"94b5e918-acce-4fd9-ad4a-c37d87965667\" (UID: \"94b5e918-acce-4fd9-ad4a-c37d87965667\") " Nov 03 20:33:47.651965 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.651935 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-socket" pod "94b5e918-acce-4fd9-ad4a-c37d87965667" (UID: "94b5e918-acce-4fd9-ad4a-c37d87965667"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.652741 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.652163 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.652741 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.652177 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-credential-socket" pod "94b5e918-acce-4fd9-ad4a-c37d87965667" (UID: "94b5e918-acce-4fd9-ad4a-c37d87965667"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.652741 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.652375 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-certs" pod "94b5e918-acce-4fd9-ad4a-c37d87965667" (UID: "94b5e918-acce-4fd9-ad4a-c37d87965667"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.653010 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.652858 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-data" pod "94b5e918-acce-4fd9-ad4a-c37d87965667" (UID: "94b5e918-acce-4fd9-ad4a-c37d87965667"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.653236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.653202 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b5e918-acce-4fd9-ad4a-c37d87965667-istiod-ca-cert" pod "94b5e918-acce-4fd9-ad4a-c37d87965667" (UID: "94b5e918-acce-4fd9-ad4a-c37d87965667"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:47.664440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.658835 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-token" pod "94b5e918-acce-4fd9-ad4a-c37d87965667" (UID: "94b5e918-acce-4fd9-ad4a-c37d87965667"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:47.664440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.659180 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-envoy" pod "94b5e918-acce-4fd9-ad4a-c37d87965667" (UID: "94b5e918-acce-4fd9-ad4a-c37d87965667"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:33:47.669541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.669486 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-kube-api-access-vp4zs" pod "94b5e918-acce-4fd9-ad4a-c37d87965667" (UID: "94b5e918-acce-4fd9-ad4a-c37d87965667"). InnerVolumeSpecName "kube-api-access-vp4zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:47.670776 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.670749 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-podinfo" pod "94b5e918-acce-4fd9-ad4a-c37d87965667" (UID: "94b5e918-acce-4fd9-ad4a-c37d87965667"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:33:47.753750 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.753719 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.753750 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.753747 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b5e918-acce-4fd9-ad4a-c37d87965667-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.753987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.753759 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp4zs\" (UniqueName: \"kubernetes.io/projected/94b5e918-acce-4fd9-ad4a-c37d87965667-kube-api-access-vp4zs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.753987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.753771 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.753987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.753780 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.753987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.753789 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.753987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.753802 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.753987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.753811 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b5e918-acce-4fd9-ad4a-c37d87965667-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:47.760698 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.760669 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" path="/var/lib/kubelet/pods/06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18/volumes" Nov 03 20:33:47.761170 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.761155 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d44a0d-26ce-4622-b30f-f2c6e013828b" path="/var/lib/kubelet/pods/38d44a0d-26ce-4622-b30f-f2c6e013828b/volumes" Nov 03 20:33:47.761597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.761579 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c89239-0110-425d-9aff-2652c9b8dfd7" path="/var/lib/kubelet/pods/61c89239-0110-425d-9aff-2652c9b8dfd7/volumes" Nov 03 20:33:47.954325 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.954219 4457 generic.go:356] "Generic (PLEG): container finished" podID="94b5e918-acce-4fd9-ad4a-c37d87965667" containerID="7c4d688bf3e753541e2e7efe52d0579b9c3086d36814b5657b4f4339b86184df" exitCode=0 Nov 03 20:33:47.954325 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.954315 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" Nov 03 20:33:47.954607 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.954359 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" event={"ID":"94b5e918-acce-4fd9-ad4a-c37d87965667","Type":"ContainerDied","Data":"7c4d688bf3e753541e2e7efe52d0579b9c3086d36814b5657b4f4339b86184df"} Nov 03 20:33:47.954607 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.954487 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26" event={"ID":"94b5e918-acce-4fd9-ad4a-c37d87965667","Type":"ContainerDied","Data":"a6e9fc6435397fe44d886fffe9ddba733a4369044f267e04f1e40afbc124c316"} Nov 03 20:33:47.954607 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.954539 4457 scope.go:122] "RemoveContainer" containerID="7c4d688bf3e753541e2e7efe52d0579b9c3086d36814b5657b4f4339b86184df" Nov 03 20:33:47.968754 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.968736 4457 scope.go:122] "RemoveContainer" containerID="7c4d688bf3e753541e2e7efe52d0579b9c3086d36814b5657b4f4339b86184df" Nov 03 20:33:47.969070 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:47.969048 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4d688bf3e753541e2e7efe52d0579b9c3086d36814b5657b4f4339b86184df\": container with ID starting with 7c4d688bf3e753541e2e7efe52d0579b9c3086d36814b5657b4f4339b86184df not found: ID does not exist" containerID="7c4d688bf3e753541e2e7efe52d0579b9c3086d36814b5657b4f4339b86184df" Nov 03 20:33:47.969125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.969083 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4d688bf3e753541e2e7efe52d0579b9c3086d36814b5657b4f4339b86184df"} err="failed to get container status \"7c4d688bf3e753541e2e7efe52d0579b9c3086d36814b5657b4f4339b86184df\": rpc error: code = NotFound desc = could not find container \"7c4d688bf3e753541e2e7efe52d0579b9c3086d36814b5657b4f4339b86184df\": container with ID starting with 7c4d688bf3e753541e2e7efe52d0579b9c3086d36814b5657b4f4339b86184df not found: ID does not exist" Nov 03 20:33:47.990754 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.990723 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26"] Nov 03 20:33:47.995269 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:47.995246 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-90ef9c-openshift-default-7f69c9cc49-slp26"] Nov 03 20:33:49.760563 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:49.760526 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b5e918-acce-4fd9-ad4a-c37d87965667" path="/var/lib/kubelet/pods/94b5e918-acce-4fd9-ad4a-c37d87965667/volumes" Nov 03 20:33:50.742230 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.742192 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4"] Nov 03 20:33:50.743181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743159 4457 cpu_manager.go:414] "containerMap: removing container" podUID="7e03f447-8195-4c9f-a9b5-6dd64724624a" containerName="revoke-keys" Nov 03 20:33:50.743259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743184 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e03f447-8195-4c9f-a9b5-6dd64724624a" containerName="revoke-keys" Nov 03 20:33:50.743259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743202 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5" containerName="payload-processing" Nov 03 20:33:50.743259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743210 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5" containerName="payload-processing" Nov 03 20:33:50.743259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743227 4457 cpu_manager.go:414] "containerMap: removing container" podUID="61c89239-0110-425d-9aff-2652c9b8dfd7" containerName="payload-pre-processing" Nov 03 20:33:50.743259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743234 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c89239-0110-425d-9aff-2652c9b8dfd7" containerName="payload-pre-processing" Nov 03 20:33:50.743259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743250 4457 cpu_manager.go:414] "containerMap: removing container" podUID="7b3116a9-eaba-42f2-a977-082a8228165b" containerName="payload-pre-processing" Nov 03 20:33:50.743259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743258 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3116a9-eaba-42f2-a977-082a8228165b" containerName="payload-pre-processing" Nov 03 20:33:50.743649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743282 4457 cpu_manager.go:414] "containerMap: removing container" podUID="94b5e918-acce-4fd9-ad4a-c37d87965667" containerName="istio-proxy" Nov 03 20:33:50.743649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743289 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b5e918-acce-4fd9-ad4a-c37d87965667" containerName="istio-proxy" Nov 03 20:33:50.743649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743304 4457 cpu_manager.go:414] "containerMap: removing container" podUID="38d44a0d-26ce-4622-b30f-f2c6e013828b" containerName="istio-proxy" Nov 03 20:33:50.743649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743311 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d44a0d-26ce-4622-b30f-f2c6e013828b" containerName="istio-proxy" Nov 03 20:33:50.743649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743321 4457 cpu_manager.go:414] "containerMap: removing container" podUID="47128d77-8e6f-4e60-9299-585cfe98b268" containerName="payload-pre-processing" Nov 03 20:33:50.743649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743329 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47128d77-8e6f-4e60-9299-585cfe98b268" containerName="payload-pre-processing" Nov 03 20:33:50.743649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743370 4457 cpu_manager.go:414] "containerMap: removing container" podUID="a9cd7625-f8fb-42aa-a39a-5ea33b4b8831" containerName="maas-api" Nov 03 20:33:50.743649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743377 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cd7625-f8fb-42aa-a39a-5ea33b4b8831" containerName="maas-api" Nov 03 20:33:50.743649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743411 4457 cpu_manager.go:414] "containerMap: removing container" podUID="06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" containerName="istio-proxy" Nov 03 20:33:50.743649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.743419 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" containerName="istio-proxy" Nov 03 20:33:50.752877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.744402 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cd7625-f8fb-42aa-a39a-5ea33b4b8831" containerName="maas-api" Nov 03 20:33:50.752877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.744432 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ca19a7-29db-4b9c-a0fd-5f75e1bf0e18" containerName="istio-proxy" Nov 03 20:33:50.752877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.744455 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e03f447-8195-4c9f-a9b5-6dd64724624a" containerName="revoke-keys" Nov 03 20:33:50.752877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.744467 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="2698e3c6-1dfd-4e2b-9d88-02dbee3f64e5" containerName="payload-processing" Nov 03 20:33:50.752877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.744487 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c89239-0110-425d-9aff-2652c9b8dfd7" containerName="payload-pre-processing" Nov 03 20:33:50.752877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.744500 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3116a9-eaba-42f2-a977-082a8228165b" containerName="payload-pre-processing" Nov 03 20:33:50.752877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.744522 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b5e918-acce-4fd9-ad4a-c37d87965667" containerName="istio-proxy" Nov 03 20:33:50.752877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.744534 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d44a0d-26ce-4622-b30f-f2c6e013828b" containerName="istio-proxy" Nov 03 20:33:50.752877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.744554 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="47128d77-8e6f-4e60-9299-585cfe98b268" containerName="payload-pre-processing" Nov 03 20:33:50.762786 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.758578 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" Nov 03 20:33:50.771934 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.771905 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4"] Nov 03 20:33:50.779145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.779118 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:33:50.779727 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.779704 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:33:50.797161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.797113 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4\" (UID: \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" Nov 03 20:33:50.797380 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.797350 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4\" (UID: \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" Nov 03 20:33:50.797522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.797502 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4622\" (UniqueName: \"kubernetes.io/projected/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-kube-api-access-s4622\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4\" (UID: \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" Nov 03 20:33:50.898651 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.898600 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4\" (UID: \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" Nov 03 20:33:50.898837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.898754 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4\" (UID: \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" Nov 03 20:33:50.898837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.898811 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4622\" (UniqueName: \"kubernetes.io/projected/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-kube-api-access-s4622\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4\" (UID: \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" Nov 03 20:33:50.899679 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.899645 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4\" (UID: \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" Nov 03 20:33:50.900030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.899997 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" podUID="7e57f420-40a0-4f1b-8b58-70f0be591ec8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:33:50.901549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.901520 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4\" (UID: \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" Nov 03 20:33:50.911298 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:50.911274 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4622\" (UniqueName: \"kubernetes.io/projected/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-kube-api-access-s4622\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4\" (UID: \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" Nov 03 20:33:51.087333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:51.087295 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" Nov 03 20:33:51.437905 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:33:51.437837 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4b3f08_f817_4852_88ee_6c8f3ea444e7.slice/crio-f19abf9ad122fc4168332b9eb148448a2a39355ca29d275ff0b125570def33de WatchSource:0}: Error finding container f19abf9ad122fc4168332b9eb148448a2a39355ca29d275ff0b125570def33de: Status 404 returned error can't find the container with id f19abf9ad122fc4168332b9eb148448a2a39355ca29d275ff0b125570def33de Nov 03 20:33:51.494410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:51.488719 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4"] Nov 03 20:33:52.008908 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:52.008867 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" event={"ID":"ec4b3f08-f817-4852-88ee-6c8f3ea444e7","Type":"ContainerStarted","Data":"b78bbca99cdfe6e5da1293e64f0389aeae8836f57c30ae21143e6bf2df03ec7d"} Nov 03 20:33:52.009436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:52.008931 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" event={"ID":"ec4b3f08-f817-4852-88ee-6c8f3ea444e7","Type":"ContainerStarted","Data":"f19abf9ad122fc4168332b9eb148448a2a39355ca29d275ff0b125570def33de"} Nov 03 20:33:52.029066 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:52.029008 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" podStartSLOduration=2.02899335 podStartE2EDuration="2.02899335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:33:52.026939681 +0000 UTC m=+2210.402536430" watchObservedRunningTime="2026-11-03 20:33:52.02899335 +0000 UTC m=+2210.404590093" Nov 03 20:33:53.019333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:53.019237 4457 generic.go:356] "Generic (PLEG): container finished" podID="ec4b3f08-f817-4852-88ee-6c8f3ea444e7" containerID="b78bbca99cdfe6e5da1293e64f0389aeae8836f57c30ae21143e6bf2df03ec7d" exitCode=0 Nov 03 20:33:53.019333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:53.019320 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" event={"ID":"ec4b3f08-f817-4852-88ee-6c8f3ea444e7","Type":"ContainerDied","Data":"b78bbca99cdfe6e5da1293e64f0389aeae8836f57c30ae21143e6bf2df03ec7d"} Nov 03 20:33:54.222361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:54.222333 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" Nov 03 20:33:54.369261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:54.369224 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-xks-ca\") pod \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\" (UID: \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\") " Nov 03 20:33:54.369505 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:54.369430 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-kube-api-access-s4622\" (UniqueName: \"kubernetes.io/projected/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-kube-api-access-s4622\") pod \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\" (UID: \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\") " Nov 03 20:33:54.369597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:54.369572 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-openshift-ca\") pod \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\" (UID: \"ec4b3f08-f817-4852-88ee-6c8f3ea444e7\") " Nov 03 20:33:54.369912 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:54.369842 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-openshift-ca" pod "ec4b3f08-f817-4852-88ee-6c8f3ea444e7" (UID: "ec4b3f08-f817-4852-88ee-6c8f3ea444e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:54.370538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:54.370517 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:54.371510 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:54.371488 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-xks-ca" pod "ec4b3f08-f817-4852-88ee-6c8f3ea444e7" (UID: "ec4b3f08-f817-4852-88ee-6c8f3ea444e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:33:54.372030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:54.371979 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-kube-api-access-s4622" pod "ec4b3f08-f817-4852-88ee-6c8f3ea444e7" (UID: "ec4b3f08-f817-4852-88ee-6c8f3ea444e7"). InnerVolumeSpecName "kube-api-access-s4622". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:54.471964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:54.471926 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:54.471964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:54.471961 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4622\" (UniqueName: \"kubernetes.io/projected/ec4b3f08-f817-4852-88ee-6c8f3ea444e7-kube-api-access-s4622\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:55.044513 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:55.044468 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" event={"ID":"ec4b3f08-f817-4852-88ee-6c8f3ea444e7","Type":"ContainerDied","Data":"f19abf9ad122fc4168332b9eb148448a2a39355ca29d275ff0b125570def33de"} Nov 03 20:33:55.044724 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:55.044586 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f19abf9ad122fc4168332b9eb148448a2a39355ca29d275ff0b125570def33de" Nov 03 20:33:55.044724 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:55.044686 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4" Nov 03 20:33:58.682767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:58.682738 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" Nov 03 20:33:58.753323 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:58.753290 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e57f420-40a0-4f1b-8b58-70f0be591ec8-kube-api-access-2mj5t\" (UniqueName: \"kubernetes.io/projected/7e57f420-40a0-4f1b-8b58-70f0be591ec8-kube-api-access-2mj5t\") pod \"7e57f420-40a0-4f1b-8b58-70f0be591ec8\" (UID: \"7e57f420-40a0-4f1b-8b58-70f0be591ec8\") " Nov 03 20:33:58.753540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:58.753358 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e57f420-40a0-4f1b-8b58-70f0be591ec8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e57f420-40a0-4f1b-8b58-70f0be591ec8-plugins-config-volume\") pod \"7e57f420-40a0-4f1b-8b58-70f0be591ec8\" (UID: \"7e57f420-40a0-4f1b-8b58-70f0be591ec8\") " Nov 03 20:33:58.753924 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:58.753895 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e57f420-40a0-4f1b-8b58-70f0be591ec8-plugins-config-volume" pod "7e57f420-40a0-4f1b-8b58-70f0be591ec8" (UID: "7e57f420-40a0-4f1b-8b58-70f0be591ec8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:33:58.755656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:58.755625 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e57f420-40a0-4f1b-8b58-70f0be591ec8-kube-api-access-2mj5t" pod "7e57f420-40a0-4f1b-8b58-70f0be591ec8" (UID: "7e57f420-40a0-4f1b-8b58-70f0be591ec8"). InnerVolumeSpecName "kube-api-access-2mj5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:33:58.855085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:58.855050 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mj5t\" (UniqueName: \"kubernetes.io/projected/7e57f420-40a0-4f1b-8b58-70f0be591ec8-kube-api-access-2mj5t\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:58.855251 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:58.855094 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e57f420-40a0-4f1b-8b58-70f0be591ec8-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:33:59.087829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:59.087780 4457 generic.go:356] "Generic (PLEG): container finished" podID="7e57f420-40a0-4f1b-8b58-70f0be591ec8" containerID="d20c1f13b2783e6999c0019d611fed0d580684674f82bf2c36372d162d9bcd01" exitCode=0 Nov 03 20:33:59.087829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:59.087827 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" Nov 03 20:33:59.088082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:59.087846 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" event={"ID":"7e57f420-40a0-4f1b-8b58-70f0be591ec8","Type":"ContainerDied","Data":"d20c1f13b2783e6999c0019d611fed0d580684674f82bf2c36372d162d9bcd01"} Nov 03 20:33:59.088082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:59.087900 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq" event={"ID":"7e57f420-40a0-4f1b-8b58-70f0be591ec8","Type":"ContainerDied","Data":"54db25bd8e26be88d31f2bf23caf2958e74247487549d808526e2ed06376a7c3"} Nov 03 20:33:59.088082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:59.087925 4457 scope.go:122] "RemoveContainer" containerID="d20c1f13b2783e6999c0019d611fed0d580684674f82bf2c36372d162d9bcd01" Nov 03 20:33:59.115730 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:59.115703 4457 scope.go:122] "RemoveContainer" containerID="d20c1f13b2783e6999c0019d611fed0d580684674f82bf2c36372d162d9bcd01" Nov 03 20:33:59.116097 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:33:59.116074 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20c1f13b2783e6999c0019d611fed0d580684674f82bf2c36372d162d9bcd01\": container with ID starting with d20c1f13b2783e6999c0019d611fed0d580684674f82bf2c36372d162d9bcd01 not found: ID does not exist" containerID="d20c1f13b2783e6999c0019d611fed0d580684674f82bf2c36372d162d9bcd01" Nov 03 20:33:59.116164 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:59.116104 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20c1f13b2783e6999c0019d611fed0d580684674f82bf2c36372d162d9bcd01"} err="failed to get container status \"d20c1f13b2783e6999c0019d611fed0d580684674f82bf2c36372d162d9bcd01\": rpc error: code = NotFound desc = could not find container \"d20c1f13b2783e6999c0019d611fed0d580684674f82bf2c36372d162d9bcd01\": container with ID starting with d20c1f13b2783e6999c0019d611fed0d580684674f82bf2c36372d162d9bcd01 not found: ID does not exist" Nov 03 20:33:59.119374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:59.119318 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq"] Nov 03 20:33:59.123347 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:59.123325 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b0adda-747f567fcf-8wzpq"] Nov 03 20:33:59.762419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:59.762364 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e57f420-40a0-4f1b-8b58-70f0be591ec8" path="/var/lib/kubelet/pods/7e57f420-40a0-4f1b-8b58-70f0be591ec8/volumes" Nov 03 20:33:59.929345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:59.929261 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm"] Nov 03 20:33:59.929557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:33:59.929529 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" podUID="b1dfce2c-4a66-479e-afcf-ddce81a9252b" containerName="maas-api" containerID="cri-o://9d32303bd0b4291937f0df403840ef93fd12f5181839b0038ae162cb619e156b" gracePeriod=30 Nov 03 20:34:00.103412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.102979 4457 generic.go:356] "Generic (PLEG): container finished" podID="b1dfce2c-4a66-479e-afcf-ddce81a9252b" containerID="9d32303bd0b4291937f0df403840ef93fd12f5181839b0038ae162cb619e156b" exitCode=0 Nov 03 20:34:00.103412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.103117 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" event={"ID":"b1dfce2c-4a66-479e-afcf-ddce81a9252b","Type":"ContainerDied","Data":"9d32303bd0b4291937f0df403840ef93fd12f5181839b0038ae162cb619e156b"} Nov 03 20:34:00.406768 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.406739 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:34:00.506502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.506458 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1dfce2c-4a66-479e-afcf-ddce81a9252b-kube-api-access-5kzf2\" (UniqueName: \"kubernetes.io/projected/b1dfce2c-4a66-479e-afcf-ddce81a9252b-kube-api-access-5kzf2\") pod \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " Nov 03 20:34:00.506704 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.506646 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-maas-api-tls\") pod \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " Nov 03 20:34:00.506803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.506754 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1dfce2c-4a66-479e-afcf-ddce81a9252b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dfce2c-4a66-479e-afcf-ddce81a9252b-openshift-service-ca\") pod \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " Nov 03 20:34:00.506803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.506787 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-metrics-certs\") pod \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\" (UID: \"b1dfce2c-4a66-479e-afcf-ddce81a9252b\") " Nov 03 20:34:00.507208 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.507174 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1dfce2c-4a66-479e-afcf-ddce81a9252b-openshift-service-ca" pod "b1dfce2c-4a66-479e-afcf-ddce81a9252b" (UID: "b1dfce2c-4a66-479e-afcf-ddce81a9252b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:00.507399 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.507364 4457 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1dfce2c-4a66-479e-afcf-ddce81a9252b-openshift-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:00.509024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.508993 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-metrics-certs" pod "b1dfce2c-4a66-479e-afcf-ddce81a9252b" (UID: "b1dfce2c-4a66-479e-afcf-ddce81a9252b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:00.509024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.509008 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1dfce2c-4a66-479e-afcf-ddce81a9252b-kube-api-access-5kzf2" pod "b1dfce2c-4a66-479e-afcf-ddce81a9252b" (UID: "b1dfce2c-4a66-479e-afcf-ddce81a9252b"). InnerVolumeSpecName "kube-api-access-5kzf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:00.509177 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.509158 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-maas-api-tls" pod "b1dfce2c-4a66-479e-afcf-ddce81a9252b" (UID: "b1dfce2c-4a66-479e-afcf-ddce81a9252b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:00.608843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.608809 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-maas-api-tls\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:00.608843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.608840 4457 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1dfce2c-4a66-479e-afcf-ddce81a9252b-metrics-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:00.608843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.608850 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kzf2\" (UniqueName: \"kubernetes.io/projected/b1dfce2c-4a66-479e-afcf-ddce81a9252b-kube-api-access-5kzf2\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:00.927572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.927532 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7"] Nov 03 20:34:00.942416 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:00.942352 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b"] Nov 03 20:34:01.115940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:01.115907 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" Nov 03 20:34:01.116140 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:01.115941 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm" event={"ID":"b1dfce2c-4a66-479e-afcf-ddce81a9252b","Type":"ContainerDied","Data":"f9ec585a087f49a1a7dfd812a0dfb0826e03b5c93d9750436f648b195771b497"} Nov 03 20:34:01.116140 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:01.116012 4457 scope.go:122] "RemoveContainer" containerID="9d32303bd0b4291937f0df403840ef93fd12f5181839b0038ae162cb619e156b" Nov 03 20:34:01.142275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:01.142247 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm"] Nov 03 20:34:01.148872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:01.148846 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-b35c30-7b5d69cbc8-26clm"] Nov 03 20:34:01.759903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:01.759869 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1dfce2c-4a66-479e-afcf-ddce81a9252b" path="/var/lib/kubelet/pods/b1dfce2c-4a66-479e-afcf-ddce81a9252b/volumes" Nov 03 20:34:02.474557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.474469 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4"] Nov 03 20:34:02.505906 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.505870 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ba8f5b-9ntr4"] Nov 03 20:34:02.539862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.539832 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8"] Nov 03 20:34:02.540508 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.540491 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ec4b3f08-f817-4852-88ee-6c8f3ea444e7" containerName="revoke-keys" Nov 03 20:34:02.540566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.540509 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4b3f08-f817-4852-88ee-6c8f3ea444e7" containerName="revoke-keys" Nov 03 20:34:02.540566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.540542 4457 cpu_manager.go:414] "containerMap: removing container" podUID="7e57f420-40a0-4f1b-8b58-70f0be591ec8" containerName="payload-pre-processing" Nov 03 20:34:02.540566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.540548 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e57f420-40a0-4f1b-8b58-70f0be591ec8" containerName="payload-pre-processing" Nov 03 20:34:02.540666 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.540567 4457 cpu_manager.go:414] "containerMap: removing container" podUID="b1dfce2c-4a66-479e-afcf-ddce81a9252b" containerName="maas-api" Nov 03 20:34:02.540666 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.540573 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1dfce2c-4a66-479e-afcf-ddce81a9252b" containerName="maas-api" Nov 03 20:34:02.540728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.540667 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4b3f08-f817-4852-88ee-6c8f3ea444e7" containerName="revoke-keys" Nov 03 20:34:02.540728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.540681 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e57f420-40a0-4f1b-8b58-70f0be591ec8" containerName="payload-pre-processing" Nov 03 20:34:02.540728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.540690 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1dfce2c-4a66-479e-afcf-ddce81a9252b" containerName="maas-api" Nov 03 20:34:02.546074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.546052 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" Nov 03 20:34:02.553650 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.553616 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8"] Nov 03 20:34:02.559686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.559624 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:34:02.559958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.559941 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:34:02.567157 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.567091 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8\" (UID: \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" Nov 03 20:34:02.567300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.567264 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ks9b\" (UniqueName: \"kubernetes.io/projected/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-kube-api-access-7ks9b\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8\" (UID: \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" Nov 03 20:34:02.567378 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.567341 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8\" (UID: \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" Nov 03 20:34:02.669530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.669492 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8\" (UID: \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" Nov 03 20:34:02.669736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.669629 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ks9b\" (UniqueName: \"kubernetes.io/projected/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-kube-api-access-7ks9b\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8\" (UID: \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" Nov 03 20:34:02.669888 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.669746 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8\" (UID: \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" Nov 03 20:34:02.671446 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.670503 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8\" (UID: \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" Nov 03 20:34:02.672849 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.672818 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8\" (UID: \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" Nov 03 20:34:02.682581 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.682549 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ks9b\" (UniqueName: \"kubernetes.io/projected/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-kube-api-access-7ks9b\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8\" (UID: \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" Nov 03 20:34:02.870599 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.870562 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" Nov 03 20:34:02.984262 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:02.983961 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb"] Nov 03 20:34:03.215234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.215150 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8"] Nov 03 20:34:03.216469 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:34:03.216441 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e54de9_0e2f_4ae1_88ad_1ad3298c3cdf.slice/crio-87ee892affde8b836ebf221dc79d6733fd9589b58f4be21ccca40d5f3e447420 WatchSource:0}: Error finding container 87ee892affde8b836ebf221dc79d6733fd9589b58f4be21ccca40d5f3e447420: Status 404 returned error can't find the container with id 87ee892affde8b836ebf221dc79d6733fd9589b58f4be21ccca40d5f3e447420 Nov 03 20:34:03.443995 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.443960 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl"] Nov 03 20:34:03.449333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.449312 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" Nov 03 20:34:03.463427 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.461167 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl"] Nov 03 20:34:03.504081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.504039 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499b0549-6rctl\" (UID: \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" Nov 03 20:34:03.504597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.504092 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499b0549-6rctl\" (UID: \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" Nov 03 20:34:03.504597 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.504241 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skb24\" (UniqueName: \"kubernetes.io/projected/67bd4112-42a5-485f-9a4f-279ffcda0d4e-kube-api-access-skb24\") pod \"maas-api-revoke-sub-499b0549-6rctl\" (UID: \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" Nov 03 20:34:03.605350 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.605317 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skb24\" (UniqueName: \"kubernetes.io/projected/67bd4112-42a5-485f-9a4f-279ffcda0d4e-kube-api-access-skb24\") pod \"maas-api-revoke-sub-499b0549-6rctl\" (UID: \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" Nov 03 20:34:03.605553 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.605451 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499b0549-6rctl\" (UID: \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" Nov 03 20:34:03.605553 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.605475 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499b0549-6rctl\" (UID: \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" Nov 03 20:34:03.606124 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.606093 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-499b0549-6rctl\" (UID: \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" Nov 03 20:34:03.607783 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.607761 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-499b0549-6rctl\" (UID: \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" Nov 03 20:34:03.622974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.622947 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skb24\" (UniqueName: \"kubernetes.io/projected/67bd4112-42a5-485f-9a4f-279ffcda0d4e-kube-api-access-skb24\") pod \"maas-api-revoke-sub-499b0549-6rctl\" (UID: \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" Nov 03 20:34:03.761147 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.761051 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4b3f08-f817-4852-88ee-6c8f3ea444e7" path="/var/lib/kubelet/pods/ec4b3f08-f817-4852-88ee-6c8f3ea444e7/volumes" Nov 03 20:34:03.768504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:03.768477 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" Nov 03 20:34:04.064359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:04.064325 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl"] Nov 03 20:34:04.068794 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:34:04.068756 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bd4112_42a5_485f_9a4f_279ffcda0d4e.slice/crio-e2dc5dbea3ba61ce3f110d4f4434bfc94c406fc5f5a82d2dbb74191e83f0c751 WatchSource:0}: Error finding container e2dc5dbea3ba61ce3f110d4f4434bfc94c406fc5f5a82d2dbb74191e83f0c751: Status 404 returned error can't find the container with id e2dc5dbea3ba61ce3f110d4f4434bfc94c406fc5f5a82d2dbb74191e83f0c751 Nov 03 20:34:04.206826 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:04.206792 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" event={"ID":"67bd4112-42a5-485f-9a4f-279ffcda0d4e","Type":"ContainerStarted","Data":"e2dc5dbea3ba61ce3f110d4f4434bfc94c406fc5f5a82d2dbb74191e83f0c751"} Nov 03 20:34:04.208639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:04.208603 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" event={"ID":"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf","Type":"ContainerStarted","Data":"81495a6689d358d2a7de9eb6f4124b6e4a8be31640f2e5961a37323d26ec0d9d"} Nov 03 20:34:04.208772 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:04.208662 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" event={"ID":"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf","Type":"ContainerStarted","Data":"87ee892affde8b836ebf221dc79d6733fd9589b58f4be21ccca40d5f3e447420"} Nov 03 20:34:04.232175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:04.232123 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" podStartSLOduration=2.232109786 podStartE2EDuration="2.232109786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:04.229294401 +0000 UTC m=+2222.604891149" watchObservedRunningTime="2026-11-03 20:34:04.232109786 +0000 UTC m=+2222.607706532" Nov 03 20:34:05.219160 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:05.219123 4457 generic.go:356] "Generic (PLEG): container finished" podID="67bd4112-42a5-485f-9a4f-279ffcda0d4e" containerID="3113a3c3858a888be405ef967dce24272171dd8451e7635de5469367453d0576" exitCode=0 Nov 03 20:34:05.219649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:05.219201 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" event={"ID":"67bd4112-42a5-485f-9a4f-279ffcda0d4e","Type":"ContainerDied","Data":"3113a3c3858a888be405ef967dce24272171dd8451e7635de5469367453d0576"} Nov 03 20:34:05.221053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:05.221027 4457 generic.go:356] "Generic (PLEG): container finished" podID="83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf" containerID="81495a6689d358d2a7de9eb6f4124b6e4a8be31640f2e5961a37323d26ec0d9d" exitCode=0 Nov 03 20:34:05.221173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:05.221108 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" event={"ID":"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf","Type":"ContainerDied","Data":"81495a6689d358d2a7de9eb6f4124b6e4a8be31640f2e5961a37323d26ec0d9d"} Nov 03 20:34:05.928562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:05.928502 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" podUID="ead54f07-35d4-49da-b436-54f0421b6261" containerName="payload-processing" containerID="cri-o://c1c61ed92b2a4326fd297251a2475e7b97cb0b7e89701addfa4ea01d65fd837b" gracePeriod=25 Nov 03 20:34:05.942834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:05.942801 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" podUID="1a6ffae7-5a36-4221-8263-e56842c7d0ea" containerName="payload-pre-processing" containerID="cri-o://26483bd49bc66771e0aadeea8d84d3236fae149edb9739c7e6b12468eb692804" gracePeriod=25 Nov 03 20:34:06.191546 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:06.188731 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:34:06.240984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.238760 4457 generic.go:356] "Generic (PLEG): container finished" podID="1a6ffae7-5a36-4221-8263-e56842c7d0ea" containerID="26483bd49bc66771e0aadeea8d84d3236fae149edb9739c7e6b12468eb692804" exitCode=0 Nov 03 20:34:06.240984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.238985 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" event={"ID":"1a6ffae7-5a36-4221-8263-e56842c7d0ea","Type":"ContainerDied","Data":"26483bd49bc66771e0aadeea8d84d3236fae149edb9739c7e6b12468eb692804"} Nov 03 20:34:06.240984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.239041 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" event={"ID":"1a6ffae7-5a36-4221-8263-e56842c7d0ea","Type":"ContainerDied","Data":"8d2b60a792e4fbec0a1ca9b5004a9333126f3cadf733c0c3de0570b6f34bd492"} Nov 03 20:34:06.240984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.239066 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d2b60a792e4fbec0a1ca9b5004a9333126f3cadf733c0c3de0570b6f34bd492" Nov 03 20:34:06.241630 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.241558 4457 generic.go:356] "Generic (PLEG): container finished" podID="ead54f07-35d4-49da-b436-54f0421b6261" containerID="c1c61ed92b2a4326fd297251a2475e7b97cb0b7e89701addfa4ea01d65fd837b" exitCode=0 Nov 03 20:34:06.241929 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.241905 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" event={"ID":"ead54f07-35d4-49da-b436-54f0421b6261","Type":"ContainerDied","Data":"c1c61ed92b2a4326fd297251a2475e7b97cb0b7e89701addfa4ea01d65fd837b"} Nov 03 20:34:06.251824 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.251797 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" Nov 03 20:34:06.332562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.332529 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" Nov 03 20:34:06.347294 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.346366 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a6ffae7-5a36-4221-8263-e56842c7d0ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6ffae7-5a36-4221-8263-e56842c7d0ea-plugins-config-volume\") pod \"1a6ffae7-5a36-4221-8263-e56842c7d0ea\" (UID: \"1a6ffae7-5a36-4221-8263-e56842c7d0ea\") " Nov 03 20:34:06.347294 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.346548 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6ffae7-5a36-4221-8263-e56842c7d0ea-kube-api-access-7ffz7\" (UniqueName: \"kubernetes.io/projected/1a6ffae7-5a36-4221-8263-e56842c7d0ea-kube-api-access-7ffz7\") pod \"1a6ffae7-5a36-4221-8263-e56842c7d0ea\" (UID: \"1a6ffae7-5a36-4221-8263-e56842c7d0ea\") " Nov 03 20:34:06.349985 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.349940 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6ffae7-5a36-4221-8263-e56842c7d0ea-plugins-config-volume" pod "1a6ffae7-5a36-4221-8263-e56842c7d0ea" (UID: "1a6ffae7-5a36-4221-8263-e56842c7d0ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:06.352504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.352480 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6ffae7-5a36-4221-8263-e56842c7d0ea-kube-api-access-7ffz7" pod "1a6ffae7-5a36-4221-8263-e56842c7d0ea" (UID: "1a6ffae7-5a36-4221-8263-e56842c7d0ea"). InnerVolumeSpecName "kube-api-access-7ffz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:06.454660 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.450635 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead54f07-35d4-49da-b436-54f0421b6261-kube-api-access-nz9q9\" (UniqueName: \"kubernetes.io/projected/ead54f07-35d4-49da-b436-54f0421b6261-kube-api-access-nz9q9\") pod \"ead54f07-35d4-49da-b436-54f0421b6261\" (UID: \"ead54f07-35d4-49da-b436-54f0421b6261\") " Nov 03 20:34:06.454660 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.451000 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead54f07-35d4-49da-b436-54f0421b6261-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead54f07-35d4-49da-b436-54f0421b6261-plugins-config-volume\") pod \"ead54f07-35d4-49da-b436-54f0421b6261\" (UID: \"ead54f07-35d4-49da-b436-54f0421b6261\") " Nov 03 20:34:06.454660 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.451650 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1a6ffae7-5a36-4221-8263-e56842c7d0ea-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:06.454660 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.451675 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ffz7\" (UniqueName: \"kubernetes.io/projected/1a6ffae7-5a36-4221-8263-e56842c7d0ea-kube-api-access-7ffz7\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:06.454660 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.452125 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead54f07-35d4-49da-b436-54f0421b6261-plugins-config-volume" pod "ead54f07-35d4-49da-b436-54f0421b6261" (UID: "ead54f07-35d4-49da-b436-54f0421b6261"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:06.455804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.455420 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" Nov 03 20:34:06.461756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.461712 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead54f07-35d4-49da-b436-54f0421b6261-kube-api-access-nz9q9" pod "ead54f07-35d4-49da-b436-54f0421b6261" (UID: "ead54f07-35d4-49da-b436-54f0421b6261"). InnerVolumeSpecName "kube-api-access-nz9q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:06.494844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.494811 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" Nov 03 20:34:06.553111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.553078 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-xks-ca\") pod \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\" (UID: \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\") " Nov 03 20:34:06.553311 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.553117 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-xks-ca\") pod \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\" (UID: \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\") " Nov 03 20:34:06.553311 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.553260 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67bd4112-42a5-485f-9a4f-279ffcda0d4e-kube-api-access-skb24\" (UniqueName: \"kubernetes.io/projected/67bd4112-42a5-485f-9a4f-279ffcda0d4e-kube-api-access-skb24\") pod \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\" (UID: \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\") " Nov 03 20:34:06.553504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.553317 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-openshift-ca\") pod \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\" (UID: \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\") " Nov 03 20:34:06.553504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.553382 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-kube-api-access-7ks9b\" (UniqueName: \"kubernetes.io/projected/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-kube-api-access-7ks9b\") pod \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\" (UID: \"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf\") " Nov 03 20:34:06.553601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.553524 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-openshift-ca\") pod \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\" (UID: \"67bd4112-42a5-485f-9a4f-279ffcda0d4e\") " Nov 03 20:34:06.553750 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.553722 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-openshift-ca" pod "83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf" (UID: "83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:06.554011 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.553985 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-openshift-ca" pod "67bd4112-42a5-485f-9a4f-279ffcda0d4e" (UID: "67bd4112-42a5-485f-9a4f-279ffcda0d4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:06.554239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.554217 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:06.554304 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.554247 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ead54f07-35d4-49da-b436-54f0421b6261-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:06.554304 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.554263 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz9q9\" (UniqueName: \"kubernetes.io/projected/ead54f07-35d4-49da-b436-54f0421b6261-kube-api-access-nz9q9\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:06.554304 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.554278 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:06.558457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.555634 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-xks-ca" pod "67bd4112-42a5-485f-9a4f-279ffcda0d4e" (UID: "67bd4112-42a5-485f-9a4f-279ffcda0d4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:06.558457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.555758 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-kube-api-access-7ks9b" pod "83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf" (UID: "83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf"). InnerVolumeSpecName "kube-api-access-7ks9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:06.558457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.556788 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-xks-ca" pod "83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf" (UID: "83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:06.558457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.556847 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67bd4112-42a5-485f-9a4f-279ffcda0d4e-kube-api-access-skb24" pod "67bd4112-42a5-485f-9a4f-279ffcda0d4e" (UID: "67bd4112-42a5-485f-9a4f-279ffcda0d4e"). InnerVolumeSpecName "kube-api-access-skb24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:06.655802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.655764 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:06.655802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.655793 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67bd4112-42a5-485f-9a4f-279ffcda0d4e-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:06.655802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.655803 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skb24\" (UniqueName: \"kubernetes.io/projected/67bd4112-42a5-485f-9a4f-279ffcda0d4e-kube-api-access-skb24\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:06.655802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:06.655813 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ks9b\" (UniqueName: \"kubernetes.io/projected/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf-kube-api-access-7ks9b\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:07.254918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.254830 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" Nov 03 20:34:07.254918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.254844 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl" event={"ID":"67bd4112-42a5-485f-9a4f-279ffcda0d4e","Type":"ContainerDied","Data":"e2dc5dbea3ba61ce3f110d4f4434bfc94c406fc5f5a82d2dbb74191e83f0c751"} Nov 03 20:34:07.254918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.254895 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2dc5dbea3ba61ce3f110d4f4434bfc94c406fc5f5a82d2dbb74191e83f0c751" Nov 03 20:34:07.258641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.258618 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" Nov 03 20:34:07.258787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.258644 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8" event={"ID":"83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf","Type":"ContainerDied","Data":"87ee892affde8b836ebf221dc79d6733fd9589b58f4be21ccca40d5f3e447420"} Nov 03 20:34:07.258787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.258690 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87ee892affde8b836ebf221dc79d6733fd9589b58f4be21ccca40d5f3e447420" Nov 03 20:34:07.260712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.260692 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b" Nov 03 20:34:07.264703 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.264397 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" Nov 03 20:34:07.274639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.268018 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7" event={"ID":"ead54f07-35d4-49da-b436-54f0421b6261","Type":"ContainerDied","Data":"41c14940433d0f58680a6e1fefa8aee091fc8e28404053bcc65282e02d1a48c2"} Nov 03 20:34:07.274639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.268096 4457 scope.go:122] "RemoveContainer" containerID="c1c61ed92b2a4326fd297251a2475e7b97cb0b7e89701addfa4ea01d65fd837b" Nov 03 20:34:07.317380 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.314321 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b"] Nov 03 20:34:07.321937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.321782 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ba8f5b-7545bb467f-q7r2b"] Nov 03 20:34:07.357336 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.357286 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7"] Nov 03 20:34:07.361963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.361930 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ba8f5b-7485b6b97b-4wpq7"] Nov 03 20:34:07.761582 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.761545 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6ffae7-5a36-4221-8263-e56842c7d0ea" path="/var/lib/kubelet/pods/1a6ffae7-5a36-4221-8263-e56842c7d0ea/volumes" Nov 03 20:34:07.762222 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.762202 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead54f07-35d4-49da-b436-54f0421b6261" path="/var/lib/kubelet/pods/ead54f07-35d4-49da-b436-54f0421b6261/volumes" Nov 03 20:34:07.773010 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:07.772979 4457 scope.go:122] "RemoveContainer" containerID="26483bd49bc66771e0aadeea8d84d3236fae149edb9739c7e6b12468eb692804" Nov 03 20:34:08.906060 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.906027 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj"] Nov 03 20:34:08.906693 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.906676 4457 cpu_manager.go:414] "containerMap: removing container" podUID="1a6ffae7-5a36-4221-8263-e56842c7d0ea" containerName="payload-pre-processing" Nov 03 20:34:08.906752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.906694 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6ffae7-5a36-4221-8263-e56842c7d0ea" containerName="payload-pre-processing" Nov 03 20:34:08.906752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.906718 4457 cpu_manager.go:414] "containerMap: removing container" podUID="67bd4112-42a5-485f-9a4f-279ffcda0d4e" containerName="revoke-keys" Nov 03 20:34:08.906752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.906723 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67bd4112-42a5-485f-9a4f-279ffcda0d4e" containerName="revoke-keys" Nov 03 20:34:08.906752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.906742 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ead54f07-35d4-49da-b436-54f0421b6261" containerName="payload-processing" Nov 03 20:34:08.906752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.906747 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead54f07-35d4-49da-b436-54f0421b6261" containerName="payload-processing" Nov 03 20:34:08.906917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.906755 4457 cpu_manager.go:414] "containerMap: removing container" podUID="83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf" containerName="revoke-keys" Nov 03 20:34:08.906917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.906760 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf" containerName="revoke-keys" Nov 03 20:34:08.906917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.906853 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="67bd4112-42a5-485f-9a4f-279ffcda0d4e" containerName="revoke-keys" Nov 03 20:34:08.906917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.906865 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead54f07-35d4-49da-b436-54f0421b6261" containerName="payload-processing" Nov 03 20:34:08.906917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.906873 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf" containerName="revoke-keys" Nov 03 20:34:08.906917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.906882 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6ffae7-5a36-4221-8263-e56842c7d0ea" containerName="payload-pre-processing" Nov 03 20:34:08.912327 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.912307 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" Nov 03 20:34:08.917330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.917303 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:34:08.917627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.917602 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:34:08.922292 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:08.922250 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj"] Nov 03 20:34:09.007533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:09.007500 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgmb\" (UniqueName: \"kubernetes.io/projected/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-kube-api-access-bmgmb\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj\" (UID: \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" Nov 03 20:34:09.007700 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:09.007609 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj\" (UID: \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" Nov 03 20:34:09.007780 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:09.007765 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj\" (UID: \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" Nov 03 20:34:09.109340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:09.109299 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj\" (UID: \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" Nov 03 20:34:09.109534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:09.109469 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgmb\" (UniqueName: \"kubernetes.io/projected/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-kube-api-access-bmgmb\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj\" (UID: \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" Nov 03 20:34:09.109534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:09.109528 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj\" (UID: \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" Nov 03 20:34:09.110267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:09.110237 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj\" (UID: \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" Nov 03 20:34:09.111761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:09.111738 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj\" (UID: \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" Nov 03 20:34:09.122315 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:09.122279 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgmb\" (UniqueName: \"kubernetes.io/projected/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-kube-api-access-bmgmb\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj\" (UID: \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" Nov 03 20:34:09.236877 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:09.236788 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" Nov 03 20:34:09.547250 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:09.547212 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj"] Nov 03 20:34:09.553562 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:34:09.553372 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6a4b81_c6bc_4b20_a496_e6c36511c2f8.slice/crio-1114cc1bc7d936f1921cc77d5bf0d936fa98e20591bf6e688d0efc024557528a WatchSource:0}: Error finding container 1114cc1bc7d936f1921cc77d5bf0d936fa98e20591bf6e688d0efc024557528a: Status 404 returned error can't find the container with id 1114cc1bc7d936f1921cc77d5bf0d936fa98e20591bf6e688d0efc024557528a Nov 03 20:34:10.296196 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:10.296156 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" event={"ID":"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8","Type":"ContainerStarted","Data":"f922c9e067f054082a678628002a734b8f6498c635ee2dfbd3e29d48aaa70099"} Nov 03 20:34:10.296742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:10.296219 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" event={"ID":"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8","Type":"ContainerStarted","Data":"1114cc1bc7d936f1921cc77d5bf0d936fa98e20591bf6e688d0efc024557528a"} Nov 03 20:34:10.315812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:10.315761 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" podStartSLOduration=2.3157483660000002 podStartE2EDuration="2.315748366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:10.315200994 +0000 UTC m=+2228.690797734" watchObservedRunningTime="2026-11-03 20:34:10.315748366 +0000 UTC m=+2228.691345112" Nov 03 20:34:11.203667 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.203619 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf"] Nov 03 20:34:11.208937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.208906 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" Nov 03 20:34:11.214525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.214496 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf"] Nov 03 20:34:11.308020 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.307954 4457 generic.go:356] "Generic (PLEG): container finished" podID="bb6a4b81-c6bc-4b20-a496-e6c36511c2f8" containerID="f922c9e067f054082a678628002a734b8f6498c635ee2dfbd3e29d48aaa70099" exitCode=0 Nov 03 20:34:11.308436 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.308052 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" event={"ID":"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8","Type":"ContainerDied","Data":"f922c9e067f054082a678628002a734b8f6498c635ee2dfbd3e29d48aaa70099"} Nov 03 20:34:11.359520 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.359472 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf\" (UID: \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" Nov 03 20:34:11.359696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.359606 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf\" (UID: \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" Nov 03 20:34:11.359696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.359675 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8q5r\" (UniqueName: \"kubernetes.io/projected/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-kube-api-access-b8q5r\") pod \"maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf\" (UID: \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" Nov 03 20:34:11.461866 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.461771 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf\" (UID: \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" Nov 03 20:34:11.462032 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.461915 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf\" (UID: \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" Nov 03 20:34:11.462032 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.462019 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8q5r\" (UniqueName: \"kubernetes.io/projected/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-kube-api-access-b8q5r\") pod \"maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf\" (UID: \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" Nov 03 20:34:11.462510 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.462480 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf\" (UID: \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" Nov 03 20:34:11.464433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.464375 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf\" (UID: \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" Nov 03 20:34:11.476845 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.476813 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8q5r\" (UniqueName: \"kubernetes.io/projected/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-kube-api-access-b8q5r\") pod \"maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf\" (UID: \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" Nov 03 20:34:11.530442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.530367 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" Nov 03 20:34:11.840720 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:34:11.840682 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83cbc1d7_7635_4251_9b60_1c5a0b4bcf24.slice/crio-0714bf063b77c7aab79ce17ae68704862eabdbea0844862f4bba8417778bd1cf WatchSource:0}: Error finding container 0714bf063b77c7aab79ce17ae68704862eabdbea0844862f4bba8417778bd1cf: Status 404 returned error can't find the container with id 0714bf063b77c7aab79ce17ae68704862eabdbea0844862f4bba8417778bd1cf Nov 03 20:34:11.841136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:11.841110 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf"] Nov 03 20:34:12.318329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:12.318293 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" event={"ID":"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24","Type":"ContainerStarted","Data":"7a83d455db423f0db1ce770ff39b1a5339e69ffa2bbe9eb9dcc3dffab1805b3f"} Nov 03 20:34:12.318799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:12.318358 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" event={"ID":"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24","Type":"ContainerStarted","Data":"0714bf063b77c7aab79ce17ae68704862eabdbea0844862f4bba8417778bd1cf"} Nov 03 20:34:12.340221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:12.340171 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" podStartSLOduration=1.340157564 podStartE2EDuration="1.340157564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:12.33737299 +0000 UTC m=+2230.712969737" watchObservedRunningTime="2026-11-03 20:34:12.340157564 +0000 UTC m=+2230.715754311" Nov 03 20:34:12.522060 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:12.522034 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" Nov 03 20:34:12.688211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:12.687562 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-openshift-ca\") pod \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\" (UID: \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\") " Nov 03 20:34:12.688211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:12.687704 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-xks-ca\") pod \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\" (UID: \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\") " Nov 03 20:34:12.688211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:12.687877 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-kube-api-access-bmgmb\" (UniqueName: \"kubernetes.io/projected/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-kube-api-access-bmgmb\") pod \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\" (UID: \"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8\") " Nov 03 20:34:12.690813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:12.690750 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-openshift-ca" pod "bb6a4b81-c6bc-4b20-a496-e6c36511c2f8" (UID: "bb6a4b81-c6bc-4b20-a496-e6c36511c2f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:12.703869 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:12.703767 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-kube-api-access-bmgmb" pod "bb6a4b81-c6bc-4b20-a496-e6c36511c2f8" (UID: "bb6a4b81-c6bc-4b20-a496-e6c36511c2f8"). InnerVolumeSpecName "kube-api-access-bmgmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:12.713056 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:12.713012 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-xks-ca" pod "bb6a4b81-c6bc-4b20-a496-e6c36511c2f8" (UID: "bb6a4b81-c6bc-4b20-a496-e6c36511c2f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:12.789957 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:12.789879 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmgmb\" (UniqueName: \"kubernetes.io/projected/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-kube-api-access-bmgmb\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:12.789957 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:12.789907 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:12.789957 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:12.789923 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:13.328760 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:13.328726 4457 generic.go:356] "Generic (PLEG): container finished" podID="83cbc1d7-7635-4251-9b60-1c5a0b4bcf24" containerID="7a83d455db423f0db1ce770ff39b1a5339e69ffa2bbe9eb9dcc3dffab1805b3f" exitCode=0 Nov 03 20:34:13.329256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:13.328815 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" event={"ID":"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24","Type":"ContainerDied","Data":"7a83d455db423f0db1ce770ff39b1a5339e69ffa2bbe9eb9dcc3dffab1805b3f"} Nov 03 20:34:13.330447 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:13.330428 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" Nov 03 20:34:13.330580 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:13.330455 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj" event={"ID":"bb6a4b81-c6bc-4b20-a496-e6c36511c2f8","Type":"ContainerDied","Data":"1114cc1bc7d936f1921cc77d5bf0d936fa98e20591bf6e688d0efc024557528a"} Nov 03 20:34:13.330580 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:13.330498 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1114cc1bc7d936f1921cc77d5bf0d936fa98e20591bf6e688d0efc024557528a" Nov 03 20:34:14.018932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.018894 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r"] Nov 03 20:34:14.333489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.333451 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn"] Nov 03 20:34:14.334433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.334403 4457 cpu_manager.go:414] "containerMap: removing container" podUID="bb6a4b81-c6bc-4b20-a496-e6c36511c2f8" containerName="revoke-keys" Nov 03 20:34:14.334433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.334428 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6a4b81-c6bc-4b20-a496-e6c36511c2f8" containerName="revoke-keys" Nov 03 20:34:14.334610 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.334588 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6a4b81-c6bc-4b20-a496-e6c36511c2f8" containerName="revoke-keys" Nov 03 20:34:14.338807 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.338788 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" Nov 03 20:34:14.349244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.349221 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn"] Nov 03 20:34:14.416371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.416336 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn\" (UID: \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" Nov 03 20:34:14.416587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.416442 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwrf\" (UniqueName: \"kubernetes.io/projected/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-kube-api-access-wjwrf\") pod \"maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn\" (UID: \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" Nov 03 20:34:14.416587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.416546 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn\" (UID: \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" Nov 03 20:34:14.517450 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.517420 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn\" (UID: \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" Nov 03 20:34:14.517632 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.517582 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn\" (UID: \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" Nov 03 20:34:14.517695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.517654 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwrf\" (UniqueName: \"kubernetes.io/projected/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-kube-api-access-wjwrf\") pod \"maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn\" (UID: \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" Nov 03 20:34:14.518086 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.518063 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn\" (UID: \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" Nov 03 20:34:14.520478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.520458 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn\" (UID: \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" Nov 03 20:34:14.530260 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.530080 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwrf\" (UniqueName: \"kubernetes.io/projected/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-kube-api-access-wjwrf\") pod \"maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn\" (UID: \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" Nov 03 20:34:14.555139 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.555112 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" Nov 03 20:34:14.618330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.618247 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-kube-api-access-b8q5r\" (UniqueName: \"kubernetes.io/projected/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-kube-api-access-b8q5r\") pod \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\" (UID: \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\") " Nov 03 20:34:14.618330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.618296 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-openshift-ca\") pod \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\" (UID: \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\") " Nov 03 20:34:14.618587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.618345 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-xks-ca\") pod \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\" (UID: \"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24\") " Nov 03 20:34:14.618802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.618774 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-openshift-ca" pod "83cbc1d7-7635-4251-9b60-1c5a0b4bcf24" (UID: "83cbc1d7-7635-4251-9b60-1c5a0b4bcf24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:14.618914 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.618899 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:14.620500 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.620476 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-kube-api-access-b8q5r" pod "83cbc1d7-7635-4251-9b60-1c5a0b4bcf24" (UID: "83cbc1d7-7635-4251-9b60-1c5a0b4bcf24"). InnerVolumeSpecName "kube-api-access-b8q5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:14.620678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.620664 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-xks-ca" pod "83cbc1d7-7635-4251-9b60-1c5a0b4bcf24" (UID: "83cbc1d7-7635-4251-9b60-1c5a0b4bcf24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:14.673114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.673073 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" Nov 03 20:34:14.720659 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.720632 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8q5r\" (UniqueName: \"kubernetes.io/projected/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-kube-api-access-b8q5r\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:14.720858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.720838 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:14.968757 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:34:14.968699 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9b2b2a_f042_49c1_9a52_590f0d066a4f.slice/crio-db7454051a1a864cc97a5526ee139d8905eefa1f4a3be9c577abdbb096bc9437 WatchSource:0}: Error finding container db7454051a1a864cc97a5526ee139d8905eefa1f4a3be9c577abdbb096bc9437: Status 404 returned error can't find the container with id db7454051a1a864cc97a5526ee139d8905eefa1f4a3be9c577abdbb096bc9437 Nov 03 20:34:14.969044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:14.969006 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn"] Nov 03 20:34:15.351333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:15.351296 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" event={"ID":"2c9b2b2a-f042-49c1-9a52-590f0d066a4f","Type":"ContainerStarted","Data":"db7454051a1a864cc97a5526ee139d8905eefa1f4a3be9c577abdbb096bc9437"} Nov 03 20:34:15.352996 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:15.352967 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" Nov 03 20:34:15.353125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:15.353007 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf" event={"ID":"83cbc1d7-7635-4251-9b60-1c5a0b4bcf24","Type":"ContainerDied","Data":"0714bf063b77c7aab79ce17ae68704862eabdbea0844862f4bba8417778bd1cf"} Nov 03 20:34:15.353125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:15.353051 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0714bf063b77c7aab79ce17ae68704862eabdbea0844862f4bba8417778bd1cf" Nov 03 20:34:15.861681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:15.861646 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq"] Nov 03 20:34:15.886960 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:15.886927 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h"] Nov 03 20:34:16.363968 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:16.363933 4457 generic.go:356] "Generic (PLEG): container finished" podID="2c9b2b2a-f042-49c1-9a52-590f0d066a4f" containerID="44dcc0e7e47fd62d7897a4c25c8f0178678281c2ee10b83a3a4b738f4dec7ae1" exitCode=0 Nov 03 20:34:16.364565 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:16.364015 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" event={"ID":"2c9b2b2a-f042-49c1-9a52-590f0d066a4f","Type":"ContainerDied","Data":"44dcc0e7e47fd62d7897a4c25c8f0178678281c2ee10b83a3a4b738f4dec7ae1"} Nov 03 20:34:17.548498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:17.548472 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" Nov 03 20:34:17.559511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:17.559482 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-xks-ca\") pod \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\" (UID: \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\") " Nov 03 20:34:17.559676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:17.559578 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-kube-api-access-wjwrf\" (UniqueName: \"kubernetes.io/projected/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-kube-api-access-wjwrf\") pod \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\" (UID: \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\") " Nov 03 20:34:17.559834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:17.559815 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-openshift-ca\") pod \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\" (UID: \"2c9b2b2a-f042-49c1-9a52-590f0d066a4f\") " Nov 03 20:34:17.560382 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:17.560210 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-openshift-ca" pod "2c9b2b2a-f042-49c1-9a52-590f0d066a4f" (UID: "2c9b2b2a-f042-49c1-9a52-590f0d066a4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:17.560382 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:17.560354 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:17.579173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:17.578586 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-kube-api-access-wjwrf" pod "2c9b2b2a-f042-49c1-9a52-590f0d066a4f" (UID: "2c9b2b2a-f042-49c1-9a52-590f0d066a4f"). InnerVolumeSpecName "kube-api-access-wjwrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:17.581264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:17.580746 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-xks-ca" pod "2c9b2b2a-f042-49c1-9a52-590f0d066a4f" (UID: "2c9b2b2a-f042-49c1-9a52-590f0d066a4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:17.662168 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:17.662076 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:17.662168 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:17.662108 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjwrf\" (UniqueName: \"kubernetes.io/projected/2c9b2b2a-f042-49c1-9a52-590f0d066a4f-kube-api-access-wjwrf\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:18.038807 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.038711 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" podUID="75a384ec-5e7f-4f72-9381-92b564160ef6" containerName="main" containerID="cri-o://6333a4b6dfbff259d2434d3ea39c743cd094ba533e4a1befc379a4fc2888ac4e" gracePeriod=45 Nov 03 20:34:18.356277 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.356245 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:34:18.388939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.388643 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-home\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-home\") pod \"75a384ec-5e7f-4f72-9381-92b564160ef6\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " Nov 03 20:34:18.388939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.388719 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-tmp-dir\") pod \"75a384ec-5e7f-4f72-9381-92b564160ef6\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " Nov 03 20:34:18.388939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.388814 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-dshm\") pod \"75a384ec-5e7f-4f72-9381-92b564160ef6\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " Nov 03 20:34:18.388939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.388941 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a384ec-5e7f-4f72-9381-92b564160ef6-tls-certs\" (UniqueName: \"kubernetes.io/secret/75a384ec-5e7f-4f72-9381-92b564160ef6-tls-certs\") pod \"75a384ec-5e7f-4f72-9381-92b564160ef6\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " Nov 03 20:34:18.389296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.389035 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a384ec-5e7f-4f72-9381-92b564160ef6-kube-api-access-kzdlk\" (UniqueName: \"kubernetes.io/projected/75a384ec-5e7f-4f72-9381-92b564160ef6-kube-api-access-kzdlk\") pod \"75a384ec-5e7f-4f72-9381-92b564160ef6\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " Nov 03 20:34:18.389296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.389057 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-home" pod "75a384ec-5e7f-4f72-9381-92b564160ef6" (UID: "75a384ec-5e7f-4f72-9381-92b564160ef6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:18.389296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.389098 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-model-cache\") pod \"75a384ec-5e7f-4f72-9381-92b564160ef6\" (UID: \"75a384ec-5e7f-4f72-9381-92b564160ef6\") " Nov 03 20:34:18.389296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.389120 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-tmp-dir" pod "75a384ec-5e7f-4f72-9381-92b564160ef6" (UID: "75a384ec-5e7f-4f72-9381-92b564160ef6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:18.389658 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.389622 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:18.389658 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.389648 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:18.390082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.389843 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-model-cache" pod "75a384ec-5e7f-4f72-9381-92b564160ef6" (UID: "75a384ec-5e7f-4f72-9381-92b564160ef6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:18.391960 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.391936 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-dshm" pod "75a384ec-5e7f-4f72-9381-92b564160ef6" (UID: "75a384ec-5e7f-4f72-9381-92b564160ef6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:18.394616 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.394588 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a384ec-5e7f-4f72-9381-92b564160ef6-tls-certs" pod "75a384ec-5e7f-4f72-9381-92b564160ef6" (UID: "75a384ec-5e7f-4f72-9381-92b564160ef6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:18.394887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.394596 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a384ec-5e7f-4f72-9381-92b564160ef6-kube-api-access-kzdlk" pod "75a384ec-5e7f-4f72-9381-92b564160ef6" (UID: "75a384ec-5e7f-4f72-9381-92b564160ef6"). InnerVolumeSpecName "kube-api-access-kzdlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:18.396578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.396374 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" event={"ID":"2c9b2b2a-f042-49c1-9a52-590f0d066a4f","Type":"ContainerDied","Data":"db7454051a1a864cc97a5526ee139d8905eefa1f4a3be9c577abdbb096bc9437"} Nov 03 20:34:18.396578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.396518 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db7454051a1a864cc97a5526ee139d8905eefa1f4a3be9c577abdbb096bc9437" Nov 03 20:34:18.396962 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.396942 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn" Nov 03 20:34:18.400085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.399222 4457 generic.go:356] "Generic (PLEG): container finished" podID="75a384ec-5e7f-4f72-9381-92b564160ef6" containerID="6333a4b6dfbff259d2434d3ea39c743cd094ba533e4a1befc379a4fc2888ac4e" exitCode=0 Nov 03 20:34:18.400085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.399271 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" Nov 03 20:34:18.400085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.399308 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" event={"ID":"75a384ec-5e7f-4f72-9381-92b564160ef6","Type":"ContainerDied","Data":"6333a4b6dfbff259d2434d3ea39c743cd094ba533e4a1befc379a4fc2888ac4e"} Nov 03 20:34:18.400085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.399363 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb" event={"ID":"75a384ec-5e7f-4f72-9381-92b564160ef6","Type":"ContainerDied","Data":"fb7adf5e8c308e1ab82ae38fc23eb54e59ae726e326eac2fec8c05ab65ccf70b"} Nov 03 20:34:18.400085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.399441 4457 scope.go:122] "RemoveContainer" containerID="6333a4b6dfbff259d2434d3ea39c743cd094ba533e4a1befc379a4fc2888ac4e" Nov 03 20:34:18.414793 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.414772 4457 scope.go:122] "RemoveContainer" containerID="6333a4b6dfbff259d2434d3ea39c743cd094ba533e4a1befc379a4fc2888ac4e" Nov 03 20:34:18.415158 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:18.415135 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6333a4b6dfbff259d2434d3ea39c743cd094ba533e4a1befc379a4fc2888ac4e\": container with ID starting with 6333a4b6dfbff259d2434d3ea39c743cd094ba533e4a1befc379a4fc2888ac4e not found: ID does not exist" containerID="6333a4b6dfbff259d2434d3ea39c743cd094ba533e4a1befc379a4fc2888ac4e" Nov 03 20:34:18.415233 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.415170 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6333a4b6dfbff259d2434d3ea39c743cd094ba533e4a1befc379a4fc2888ac4e"} err="failed to get container status \"6333a4b6dfbff259d2434d3ea39c743cd094ba533e4a1befc379a4fc2888ac4e\": rpc error: code = NotFound desc = could not find container \"6333a4b6dfbff259d2434d3ea39c743cd094ba533e4a1befc379a4fc2888ac4e\": container with ID starting with 6333a4b6dfbff259d2434d3ea39c743cd094ba533e4a1befc379a4fc2888ac4e not found: ID does not exist" Nov 03 20:34:18.430044 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.430011 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb"] Nov 03 20:34:18.437248 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.437218 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c2c601/test-model-c2c601-kserve-84f4db5bcc-xlzsb"] Nov 03 20:34:18.491132 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.491081 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75a384ec-5e7f-4f72-9381-92b564160ef6-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:18.491132 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.491131 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzdlk\" (UniqueName: \"kubernetes.io/projected/75a384ec-5e7f-4f72-9381-92b564160ef6-kube-api-access-kzdlk\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:18.491358 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.491147 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:18.491358 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:18.491184 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75a384ec-5e7f-4f72-9381-92b564160ef6-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:19.760653 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:19.760618 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a384ec-5e7f-4f72-9381-92b564160ef6" path="/var/lib/kubelet/pods/75a384ec-5e7f-4f72-9381-92b564160ef6/volumes" Nov 03 20:34:21.336729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:21.336687 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t"] Nov 03 20:34:22.605795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:22.605754 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8"] Nov 03 20:34:22.610807 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:22.610698 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-3e9176-bdft8"] Nov 03 20:34:23.040125 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:23.040030 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj"] Nov 03 20:34:23.047114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:23.047080 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-ee4c78-4nmhj"] Nov 03 20:34:23.761056 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:23.761011 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf" path="/var/lib/kubelet/pods/83e54de9-0e2f-4ae1-88ad-1ad3298c3cdf/volumes" Nov 03 20:34:23.761777 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:23.761753 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6a4b81-c6bc-4b20-a496-e6c36511c2f8" path="/var/lib/kubelet/pods/bb6a4b81-c6bc-4b20-a496-e6c36511c2f8/volumes" Nov 03 20:34:24.519819 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:24.519781 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g"] Nov 03 20:34:24.542736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:24.542698 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv"] Nov 03 20:34:26.337206 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:26.337140 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" podUID="a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58" containerName="payload-processing" containerID="cri-o://93891d4861cb8f25d605df77b2b90208dbcd7462045025c23dc7224085eb9c23" gracePeriod=25 Nov 03 20:34:28.937241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.937202 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr"] Nov 03 20:34:28.937890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.937871 4457 cpu_manager.go:414] "containerMap: removing container" podUID="83cbc1d7-7635-4251-9b60-1c5a0b4bcf24" containerName="revoke-keys" Nov 03 20:34:28.937944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.937890 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83cbc1d7-7635-4251-9b60-1c5a0b4bcf24" containerName="revoke-keys" Nov 03 20:34:28.937944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.937912 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2c9b2b2a-f042-49c1-9a52-590f0d066a4f" containerName="revoke-keys" Nov 03 20:34:28.937944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.937919 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9b2b2a-f042-49c1-9a52-590f0d066a4f" containerName="revoke-keys" Nov 03 20:34:28.937944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.937934 4457 cpu_manager.go:414] "containerMap: removing container" podUID="75a384ec-5e7f-4f72-9381-92b564160ef6" containerName="main" Nov 03 20:34:28.937944 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.937939 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a384ec-5e7f-4f72-9381-92b564160ef6" containerName="main" Nov 03 20:34:28.938106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.938023 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9b2b2a-f042-49c1-9a52-590f0d066a4f" containerName="revoke-keys" Nov 03 20:34:28.938106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.938034 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a384ec-5e7f-4f72-9381-92b564160ef6" containerName="main" Nov 03 20:34:28.938106 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.938044 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="83cbc1d7-7635-4251-9b60-1c5a0b4bcf24" containerName="revoke-keys" Nov 03 20:34:28.941536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.941513 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4"] Nov 03 20:34:28.941648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.941624 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" Nov 03 20:34:28.945990 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.945964 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" Nov 03 20:34:28.949503 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.949477 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:34:28.949922 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.949861 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:34:28.981443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.962836 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr"] Nov 03 20:34:28.990366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:28.990327 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4"] Nov 03 20:34:29.020190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.020158 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9j7m\" (UniqueName: \"kubernetes.io/projected/8d43de03-9d03-4b5d-8fee-350088da28e9-kube-api-access-k9j7m\") pod \"maas-api-revoke-sub-0f9c5cd1-q6fs4\" (UID: \"8d43de03-9d03-4b5d-8fee-350088da28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" Nov 03 20:34:29.020331 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.020300 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9c5cd1-q6fs4\" (UID: \"8d43de03-9d03-4b5d-8fee-350088da28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" Nov 03 20:34:29.020451 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.020437 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d485fb8a-74nxr\" (UID: \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" Nov 03 20:34:29.020513 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.020472 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdd9p\" (UniqueName: \"kubernetes.io/projected/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-kube-api-access-wdd9p\") pod \"maas-api-revoke-sub-d485fb8a-74nxr\" (UID: \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" Nov 03 20:34:29.020628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.020614 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9c5cd1-q6fs4\" (UID: \"8d43de03-9d03-4b5d-8fee-350088da28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" Nov 03 20:34:29.020682 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.020664 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d485fb8a-74nxr\" (UID: \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" Nov 03 20:34:29.029665 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.029623 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" podUID="a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:34:29.042611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.042551 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" podUID="2a201136-0461-40e8-8388-ed9b7475223f" containerName="main" containerID="cri-o://38cfef2b0c4899cac208d8d5ef91bb86bbd7112588151a264fd2d939af8fb8b8" gracePeriod=45 Nov 03 20:34:29.121953 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.121907 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9j7m\" (UniqueName: \"kubernetes.io/projected/8d43de03-9d03-4b5d-8fee-350088da28e9-kube-api-access-k9j7m\") pod \"maas-api-revoke-sub-0f9c5cd1-q6fs4\" (UID: \"8d43de03-9d03-4b5d-8fee-350088da28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" Nov 03 20:34:29.122124 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.122016 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9c5cd1-q6fs4\" (UID: \"8d43de03-9d03-4b5d-8fee-350088da28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" Nov 03 20:34:29.122124 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.122085 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d485fb8a-74nxr\" (UID: \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" Nov 03 20:34:29.122267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.122178 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdd9p\" (UniqueName: \"kubernetes.io/projected/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-kube-api-access-wdd9p\") pod \"maas-api-revoke-sub-d485fb8a-74nxr\" (UID: \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" Nov 03 20:34:29.122317 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.122274 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9c5cd1-q6fs4\" (UID: \"8d43de03-9d03-4b5d-8fee-350088da28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" Nov 03 20:34:29.122317 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.122305 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d485fb8a-74nxr\" (UID: \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" Nov 03 20:34:29.122846 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.122824 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d485fb8a-74nxr\" (UID: \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" Nov 03 20:34:29.123033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.123012 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f9c5cd1-q6fs4\" (UID: \"8d43de03-9d03-4b5d-8fee-350088da28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" Nov 03 20:34:29.124460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.124434 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f9c5cd1-q6fs4\" (UID: \"8d43de03-9d03-4b5d-8fee-350088da28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" Nov 03 20:34:29.124710 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.124692 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d485fb8a-74nxr\" (UID: \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" Nov 03 20:34:29.134314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.134289 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdd9p\" (UniqueName: \"kubernetes.io/projected/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-kube-api-access-wdd9p\") pod \"maas-api-revoke-sub-d485fb8a-74nxr\" (UID: \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" Nov 03 20:34:29.134524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.134503 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9j7m\" (UniqueName: \"kubernetes.io/projected/8d43de03-9d03-4b5d-8fee-350088da28e9-kube-api-access-k9j7m\") pod \"maas-api-revoke-sub-0f9c5cd1-q6fs4\" (UID: \"8d43de03-9d03-4b5d-8fee-350088da28e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" Nov 03 20:34:29.291652 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.291612 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" Nov 03 20:34:29.309455 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.309422 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" Nov 03 20:34:29.361588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.360101 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:34:29.428670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.427972 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-dshm\") pod \"2a201136-0461-40e8-8388-ed9b7475223f\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " Nov 03 20:34:29.428670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.428113 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a201136-0461-40e8-8388-ed9b7475223f-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a201136-0461-40e8-8388-ed9b7475223f-tls-certs\") pod \"2a201136-0461-40e8-8388-ed9b7475223f\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " Nov 03 20:34:29.428670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.428166 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-tmp-dir\") pod \"2a201136-0461-40e8-8388-ed9b7475223f\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " Nov 03 20:34:29.428670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.428200 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a201136-0461-40e8-8388-ed9b7475223f-kube-api-access-fmp2l\" (UniqueName: \"kubernetes.io/projected/2a201136-0461-40e8-8388-ed9b7475223f-kube-api-access-fmp2l\") pod \"2a201136-0461-40e8-8388-ed9b7475223f\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " Nov 03 20:34:29.428670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.428252 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-home\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-home\") pod \"2a201136-0461-40e8-8388-ed9b7475223f\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " Nov 03 20:34:29.428670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.428273 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-model-cache\") pod \"2a201136-0461-40e8-8388-ed9b7475223f\" (UID: \"2a201136-0461-40e8-8388-ed9b7475223f\") " Nov 03 20:34:29.429516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.429272 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-model-cache" pod "2a201136-0461-40e8-8388-ed9b7475223f" (UID: "2a201136-0461-40e8-8388-ed9b7475223f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:29.429516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.429291 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-tmp-dir" pod "2a201136-0461-40e8-8388-ed9b7475223f" (UID: "2a201136-0461-40e8-8388-ed9b7475223f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:29.429516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.429489 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-home" pod "2a201136-0461-40e8-8388-ed9b7475223f" (UID: "2a201136-0461-40e8-8388-ed9b7475223f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:29.431868 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.431821 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-dshm" pod "2a201136-0461-40e8-8388-ed9b7475223f" (UID: "2a201136-0461-40e8-8388-ed9b7475223f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:29.432002 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.431887 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a201136-0461-40e8-8388-ed9b7475223f-kube-api-access-fmp2l" pod "2a201136-0461-40e8-8388-ed9b7475223f" (UID: "2a201136-0461-40e8-8388-ed9b7475223f"). InnerVolumeSpecName "kube-api-access-fmp2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:29.432002 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.431951 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a201136-0461-40e8-8388-ed9b7475223f-tls-certs" pod "2a201136-0461-40e8-8388-ed9b7475223f" (UID: "2a201136-0461-40e8-8388-ed9b7475223f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:29.520241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.520129 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" podUID="896c36e5-f4a0-479e-b3c9-7f16881b272e" containerName="payload-processing" containerID="cri-o://7e7d3c376499619d5e61271a98b04d9a2234399edaf0e6f233ad1b214c5e1255" gracePeriod=25 Nov 03 20:34:29.530502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.530476 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a201136-0461-40e8-8388-ed9b7475223f-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:29.530640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.530517 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:29.530640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.530528 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmp2l\" (UniqueName: \"kubernetes.io/projected/2a201136-0461-40e8-8388-ed9b7475223f-kube-api-access-fmp2l\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:29.530640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.530539 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:29.530640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.530547 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:29.530640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.530559 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a201136-0461-40e8-8388-ed9b7475223f-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:29.543939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.543902 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" podUID="c0e677dc-c502-43a4-9bdf-d7983a6f9b1b" containerName="payload-pre-processing" containerID="cri-o://eb792ba6fff2a1dc5eca7f82dc1ae3c95d97351e7dcc6e23f3da0898f4bd1818" gracePeriod=25 Nov 03 20:34:29.608645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.608597 4457 generic.go:356] "Generic (PLEG): container finished" podID="2a201136-0461-40e8-8388-ed9b7475223f" containerID="38cfef2b0c4899cac208d8d5ef91bb86bbd7112588151a264fd2d939af8fb8b8" exitCode=0 Nov 03 20:34:29.608645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.608644 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" Nov 03 20:34:29.608883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.608796 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" event={"ID":"2a201136-0461-40e8-8388-ed9b7475223f","Type":"ContainerDied","Data":"38cfef2b0c4899cac208d8d5ef91bb86bbd7112588151a264fd2d939af8fb8b8"} Nov 03 20:34:29.608883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.608849 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r" event={"ID":"2a201136-0461-40e8-8388-ed9b7475223f","Type":"ContainerDied","Data":"2588f0d7b07866cd2d59c9e815a98795fde9632c8cd0d4494584748bdb6e967f"} Nov 03 20:34:29.608883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.608874 4457 scope.go:122] "RemoveContainer" containerID="38cfef2b0c4899cac208d8d5ef91bb86bbd7112588151a264fd2d939af8fb8b8" Nov 03 20:34:29.621002 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.620979 4457 scope.go:122] "RemoveContainer" containerID="38cfef2b0c4899cac208d8d5ef91bb86bbd7112588151a264fd2d939af8fb8b8" Nov 03 20:34:29.621322 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:29.621301 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38cfef2b0c4899cac208d8d5ef91bb86bbd7112588151a264fd2d939af8fb8b8\": container with ID starting with 38cfef2b0c4899cac208d8d5ef91bb86bbd7112588151a264fd2d939af8fb8b8 not found: ID does not exist" containerID="38cfef2b0c4899cac208d8d5ef91bb86bbd7112588151a264fd2d939af8fb8b8" Nov 03 20:34:29.621463 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.621330 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38cfef2b0c4899cac208d8d5ef91bb86bbd7112588151a264fd2d939af8fb8b8"} err="failed to get container status \"38cfef2b0c4899cac208d8d5ef91bb86bbd7112588151a264fd2d939af8fb8b8\": rpc error: code = NotFound desc = could not find container \"38cfef2b0c4899cac208d8d5ef91bb86bbd7112588151a264fd2d939af8fb8b8\": container with ID starting with 38cfef2b0c4899cac208d8d5ef91bb86bbd7112588151a264fd2d939af8fb8b8 not found: ID does not exist" Nov 03 20:34:29.627587 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:34:29.627561 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf78616_ccd3_468d_aa28_bf3f4d98e66c.slice/crio-c99de8d94c1da028eda07b7b86472b6d7ceeb4abbc32b0aeb269ae16713c0dbb WatchSource:0}: Error finding container c99de8d94c1da028eda07b7b86472b6d7ceeb4abbc32b0aeb269ae16713c0dbb: Status 404 returned error can't find the container with id c99de8d94c1da028eda07b7b86472b6d7ceeb4abbc32b0aeb269ae16713c0dbb Nov 03 20:34:29.629429 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.629403 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr"] Nov 03 20:34:29.635431 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:34:29.635373 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d43de03_9d03_4b5d_8fee_350088da28e9.slice/crio-c654085b276047ac12db2046c9a7be5c7f8b1cd38997f30be458f702428d0b3f WatchSource:0}: Error finding container c654085b276047ac12db2046c9a7be5c7f8b1cd38997f30be458f702428d0b3f: Status 404 returned error can't find the container with id c654085b276047ac12db2046c9a7be5c7f8b1cd38997f30be458f702428d0b3f Nov 03 20:34:29.635633 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.635618 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4"] Nov 03 20:34:29.640858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.640836 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r"] Nov 03 20:34:29.645494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.645471 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a744eb/test-model-a744eb-kserve-867c49fcc7-r2w4r"] Nov 03 20:34:29.759303 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:29.759263 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a201136-0461-40e8-8388-ed9b7475223f" path="/var/lib/kubelet/pods/2a201136-0461-40e8-8388-ed9b7475223f/volumes" Nov 03 20:34:30.286523 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:30.286483 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn"] Nov 03 20:34:30.289855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:30.289821 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c2c601-k8xgn"] Nov 03 20:34:30.619276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:30.619240 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" event={"ID":"8d43de03-9d03-4b5d-8fee-350088da28e9","Type":"ContainerStarted","Data":"adbd42d72873a0730b4f71ef4302ebe69fa659c4b819ae0ece627538c395723a"} Nov 03 20:34:30.619512 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:30.619298 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" event={"ID":"8d43de03-9d03-4b5d-8fee-350088da28e9","Type":"ContainerStarted","Data":"c654085b276047ac12db2046c9a7be5c7f8b1cd38997f30be458f702428d0b3f"} Nov 03 20:34:30.622226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:30.622200 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" event={"ID":"4cf78616-ccd3-468d-aa28-bf3f4d98e66c","Type":"ContainerStarted","Data":"531bc7bc8c660b3381ae0c360063fa9606548f657d858f2b3c81ece425f9657f"} Nov 03 20:34:30.622345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:30.622254 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" event={"ID":"4cf78616-ccd3-468d-aa28-bf3f4d98e66c","Type":"ContainerStarted","Data":"c99de8d94c1da028eda07b7b86472b6d7ceeb4abbc32b0aeb269ae16713c0dbb"} Nov 03 20:34:30.637711 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:30.637660 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" podStartSLOduration=2.637644345 podStartE2EDuration="2.637644345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:30.636738269 +0000 UTC m=+2249.012335017" watchObservedRunningTime="2026-11-03 20:34:30.637644345 +0000 UTC m=+2249.013241093" Nov 03 20:34:30.656417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:30.656349 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" podStartSLOduration=2.656331926 podStartE2EDuration="2.656331926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:34:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:34:30.655218825 +0000 UTC m=+2249.030815573" watchObservedRunningTime="2026-11-03 20:34:30.656331926 +0000 UTC m=+2249.031928674" Nov 03 20:34:30.887226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:30.886999 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" containerID="cri-o://807d14e1c3eb133760224cc4d11775d5a50ca0b66655acc0adafd7b740568f07" gracePeriod=45 Nov 03 20:34:30.925931 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:30.925881 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" containerID="cri-o://1202e7c5ffe7a6e731deaed54e537188b37a2d68888eb5e3a3f4a70feba78f94" gracePeriod=45 Nov 03 20:34:31.040754 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.040719 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": dial tcp 10.133.0.79:8000: connect: connection refused" Nov 03 20:34:31.376408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.374478 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:34:31.407880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.392570 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:34:31.477179 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.477088 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tmp-dir\") pod \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " Nov 03 20:34:31.477462 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.477435 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-dshm\") pod \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " Nov 03 20:34:31.477628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.477611 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c089cbb-1f1d-4a87-97c1-b77de0012dda-kube-api-access-s9wql\" (UniqueName: \"kubernetes.io/projected/5c089cbb-1f1d-4a87-97c1-b77de0012dda-kube-api-access-s9wql\") pod \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " Nov 03 20:34:31.477802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.477784 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tls-certs\") pod \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " Nov 03 20:34:31.477984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.477966 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-tmp-dir\") pod \"5d37f893-d432-4047-8b6b-141094b1524f\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " Nov 03 20:34:31.478114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.478099 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-model-cache\") pod \"5d37f893-d432-4047-8b6b-141094b1524f\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " Nov 03 20:34:31.478257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.478238 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-dshm\") pod \"5d37f893-d432-4047-8b6b-141094b1524f\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " Nov 03 20:34:31.478476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.478457 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-model-cache\") pod \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " Nov 03 20:34:31.478700 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.478681 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d37f893-d432-4047-8b6b-141094b1524f-kube-api-access-4254p\" (UniqueName: \"kubernetes.io/projected/5d37f893-d432-4047-8b6b-141094b1524f-kube-api-access-4254p\") pod \"5d37f893-d432-4047-8b6b-141094b1524f\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " Nov 03 20:34:31.478829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.478814 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d37f893-d432-4047-8b6b-141094b1524f-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d37f893-d432-4047-8b6b-141094b1524f-tls-certs\") pod \"5d37f893-d432-4047-8b6b-141094b1524f\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " Nov 03 20:34:31.478962 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.478946 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-home\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-home\") pod \"5d37f893-d432-4047-8b6b-141094b1524f\" (UID: \"5d37f893-d432-4047-8b6b-141094b1524f\") " Nov 03 20:34:31.479127 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.479113 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-home\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-home\") pod \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\" (UID: \"5c089cbb-1f1d-4a87-97c1-b77de0012dda\") " Nov 03 20:34:31.479587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.479547 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-tmp-dir" pod "5d37f893-d432-4047-8b6b-141094b1524f" (UID: "5d37f893-d432-4047-8b6b-141094b1524f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.480161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.479755 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tmp-dir" pod "5c089cbb-1f1d-4a87-97c1-b77de0012dda" (UID: "5c089cbb-1f1d-4a87-97c1-b77de0012dda"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.480161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.480022 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-model-cache" pod "5c089cbb-1f1d-4a87-97c1-b77de0012dda" (UID: "5c089cbb-1f1d-4a87-97c1-b77de0012dda"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.480407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.480338 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-model-cache" pod "5d37f893-d432-4047-8b6b-141094b1524f" (UID: "5d37f893-d432-4047-8b6b-141094b1524f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.480624 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.480599 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-home" pod "5d37f893-d432-4047-8b6b-141094b1524f" (UID: "5d37f893-d432-4047-8b6b-141094b1524f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.480904 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.480886 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-home" pod "5c089cbb-1f1d-4a87-97c1-b77de0012dda" (UID: "5c089cbb-1f1d-4a87-97c1-b77de0012dda"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.481153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.480916 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.481153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.481066 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.481153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.481085 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.481153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.481100 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.481153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.481134 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.483328 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.483267 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c089cbb-1f1d-4a87-97c1-b77de0012dda-kube-api-access-s9wql" pod "5c089cbb-1f1d-4a87-97c1-b77de0012dda" (UID: "5c089cbb-1f1d-4a87-97c1-b77de0012dda"). InnerVolumeSpecName "kube-api-access-s9wql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:31.484079 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.484052 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d37f893-d432-4047-8b6b-141094b1524f-kube-api-access-4254p" pod "5d37f893-d432-4047-8b6b-141094b1524f" (UID: "5d37f893-d432-4047-8b6b-141094b1524f"). InnerVolumeSpecName "kube-api-access-4254p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:31.484721 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.484695 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-dshm" pod "5c089cbb-1f1d-4a87-97c1-b77de0012dda" (UID: "5c089cbb-1f1d-4a87-97c1-b77de0012dda"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.484809 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.484785 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-dshm" pod "5d37f893-d432-4047-8b6b-141094b1524f" (UID: "5d37f893-d432-4047-8b6b-141094b1524f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:31.486350 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.486329 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d37f893-d432-4047-8b6b-141094b1524f-tls-certs" pod "5d37f893-d432-4047-8b6b-141094b1524f" (UID: "5d37f893-d432-4047-8b6b-141094b1524f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:31.487225 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.487200 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tls-certs" pod "5c089cbb-1f1d-4a87-97c1-b77de0012dda" (UID: "5c089cbb-1f1d-4a87-97c1-b77de0012dda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:31.589422 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.586380 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d37f893-d432-4047-8b6b-141094b1524f-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.589422 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.586445 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4254p\" (UniqueName: \"kubernetes.io/projected/5d37f893-d432-4047-8b6b-141094b1524f-kube-api-access-4254p\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.589422 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.586464 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d37f893-d432-4047-8b6b-141094b1524f-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.589422 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.586477 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.589422 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.586489 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c089cbb-1f1d-4a87-97c1-b77de0012dda-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.589422 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.586500 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9wql\" (UniqueName: \"kubernetes.io/projected/5c089cbb-1f1d-4a87-97c1-b77de0012dda-kube-api-access-s9wql\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.589422 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.586513 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c089cbb-1f1d-4a87-97c1-b77de0012dda-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:31.705414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.704948 4457 generic.go:356] "Generic (PLEG): container finished" podID="4cf78616-ccd3-468d-aa28-bf3f4d98e66c" containerID="531bc7bc8c660b3381ae0c360063fa9606548f657d858f2b3c81ece425f9657f" exitCode=0 Nov 03 20:34:31.705414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.705030 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" event={"ID":"4cf78616-ccd3-468d-aa28-bf3f4d98e66c","Type":"ContainerDied","Data":"531bc7bc8c660b3381ae0c360063fa9606548f657d858f2b3c81ece425f9657f"} Nov 03 20:34:31.712469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.707427 4457 generic.go:356] "Generic (PLEG): container finished" podID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerID="807d14e1c3eb133760224cc4d11775d5a50ca0b66655acc0adafd7b740568f07" exitCode=0 Nov 03 20:34:31.712469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.707666 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" event={"ID":"5c089cbb-1f1d-4a87-97c1-b77de0012dda","Type":"ContainerDied","Data":"807d14e1c3eb133760224cc4d11775d5a50ca0b66655acc0adafd7b740568f07"} Nov 03 20:34:31.712469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.707711 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" event={"ID":"5c089cbb-1f1d-4a87-97c1-b77de0012dda","Type":"ContainerDied","Data":"9c2b07d66526d2f410579cb6873b1f3497e8f844f0d26a715221da472cdd49da"} Nov 03 20:34:31.712469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.707718 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq" Nov 03 20:34:31.712469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.707736 4457 scope.go:122] "RemoveContainer" containerID="807d14e1c3eb133760224cc4d11775d5a50ca0b66655acc0adafd7b740568f07" Nov 03 20:34:31.712469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.711583 4457 generic.go:356] "Generic (PLEG): container finished" podID="5d37f893-d432-4047-8b6b-141094b1524f" containerID="1202e7c5ffe7a6e731deaed54e537188b37a2d68888eb5e3a3f4a70feba78f94" exitCode=0 Nov 03 20:34:31.712469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.711742 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" event={"ID":"5d37f893-d432-4047-8b6b-141094b1524f","Type":"ContainerDied","Data":"1202e7c5ffe7a6e731deaed54e537188b37a2d68888eb5e3a3f4a70feba78f94"} Nov 03 20:34:31.712469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.711768 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" event={"ID":"5d37f893-d432-4047-8b6b-141094b1524f","Type":"ContainerDied","Data":"c6c312021c1a8a90e5f7f0d23402af51db335b937d8452b5917f085ababf25c0"} Nov 03 20:34:31.712469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.711829 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h" Nov 03 20:34:31.728971 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.728901 4457 generic.go:356] "Generic (PLEG): container finished" podID="8d43de03-9d03-4b5d-8fee-350088da28e9" containerID="adbd42d72873a0730b4f71ef4302ebe69fa659c4b819ae0ece627538c395723a" exitCode=0 Nov 03 20:34:31.729353 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.729319 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" event={"ID":"8d43de03-9d03-4b5d-8fee-350088da28e9","Type":"ContainerDied","Data":"adbd42d72873a0730b4f71ef4302ebe69fa659c4b819ae0ece627538c395723a"} Nov 03 20:34:31.736028 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.736009 4457 scope.go:122] "RemoveContainer" containerID="807d14e1c3eb133760224cc4d11775d5a50ca0b66655acc0adafd7b740568f07" Nov 03 20:34:31.736293 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:31.736265 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807d14e1c3eb133760224cc4d11775d5a50ca0b66655acc0adafd7b740568f07\": container with ID starting with 807d14e1c3eb133760224cc4d11775d5a50ca0b66655acc0adafd7b740568f07 not found: ID does not exist" containerID="807d14e1c3eb133760224cc4d11775d5a50ca0b66655acc0adafd7b740568f07" Nov 03 20:34:31.736360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.736303 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807d14e1c3eb133760224cc4d11775d5a50ca0b66655acc0adafd7b740568f07"} err="failed to get container status \"807d14e1c3eb133760224cc4d11775d5a50ca0b66655acc0adafd7b740568f07\": rpc error: code = NotFound desc = could not find container \"807d14e1c3eb133760224cc4d11775d5a50ca0b66655acc0adafd7b740568f07\": container with ID starting with 807d14e1c3eb133760224cc4d11775d5a50ca0b66655acc0adafd7b740568f07 not found: ID does not exist" Nov 03 20:34:31.736360 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.736319 4457 scope.go:122] "RemoveContainer" containerID="1202e7c5ffe7a6e731deaed54e537188b37a2d68888eb5e3a3f4a70feba78f94" Nov 03 20:34:31.750181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.750151 4457 scope.go:122] "RemoveContainer" containerID="1202e7c5ffe7a6e731deaed54e537188b37a2d68888eb5e3a3f4a70feba78f94" Nov 03 20:34:31.750925 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:31.750892 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1202e7c5ffe7a6e731deaed54e537188b37a2d68888eb5e3a3f4a70feba78f94\": container with ID starting with 1202e7c5ffe7a6e731deaed54e537188b37a2d68888eb5e3a3f4a70feba78f94 not found: ID does not exist" containerID="1202e7c5ffe7a6e731deaed54e537188b37a2d68888eb5e3a3f4a70feba78f94" Nov 03 20:34:31.751057 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.750934 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1202e7c5ffe7a6e731deaed54e537188b37a2d68888eb5e3a3f4a70feba78f94"} err="failed to get container status \"1202e7c5ffe7a6e731deaed54e537188b37a2d68888eb5e3a3f4a70feba78f94\": rpc error: code = NotFound desc = could not find container \"1202e7c5ffe7a6e731deaed54e537188b37a2d68888eb5e3a3f4a70feba78f94\": container with ID starting with 1202e7c5ffe7a6e731deaed54e537188b37a2d68888eb5e3a3f4a70feba78f94 not found: ID does not exist" Nov 03 20:34:31.767634 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.767568 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c9b2b2a-f042-49c1-9a52-590f0d066a4f" path="/var/lib/kubelet/pods/2c9b2b2a-f042-49c1-9a52-590f0d066a4f/volumes" Nov 03 20:34:31.770013 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.769985 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq"] Nov 03 20:34:31.770421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.770401 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/distinct-w3-62221d-kserve-5d5ccc7667-kr2cq"] Nov 03 20:34:31.774481 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.774455 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h"] Nov 03 20:34:31.778299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:31.778277 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-62221d/simulator-w3-62221d-kserve-547bf46c9c-qtv4h"] Nov 03 20:34:32.995317 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:32.995151 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" Nov 03 20:34:33.001849 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.001815 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" Nov 03 20:34:33.121814 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.121767 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-openshift-ca\") pod \"8d43de03-9d03-4b5d-8fee-350088da28e9\" (UID: \"8d43de03-9d03-4b5d-8fee-350088da28e9\") " Nov 03 20:34:33.122030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.121895 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-openshift-ca\") pod \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\" (UID: \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\") " Nov 03 20:34:33.122030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.121935 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-kube-api-access-wdd9p\" (UniqueName: \"kubernetes.io/projected/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-kube-api-access-wdd9p\") pod \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\" (UID: \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\") " Nov 03 20:34:33.122030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.121996 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-xks-ca\") pod \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\" (UID: \"4cf78616-ccd3-468d-aa28-bf3f4d98e66c\") " Nov 03 20:34:33.122030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.122019 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d43de03-9d03-4b5d-8fee-350088da28e9-kube-api-access-k9j7m\" (UniqueName: \"kubernetes.io/projected/8d43de03-9d03-4b5d-8fee-350088da28e9-kube-api-access-k9j7m\") pod \"8d43de03-9d03-4b5d-8fee-350088da28e9\" (UID: \"8d43de03-9d03-4b5d-8fee-350088da28e9\") " Nov 03 20:34:33.122270 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.122042 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-xks-ca\") pod \"8d43de03-9d03-4b5d-8fee-350088da28e9\" (UID: \"8d43de03-9d03-4b5d-8fee-350088da28e9\") " Nov 03 20:34:33.122270 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.122242 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-openshift-ca" pod "8d43de03-9d03-4b5d-8fee-350088da28e9" (UID: "8d43de03-9d03-4b5d-8fee-350088da28e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:33.122359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.122310 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-openshift-ca" pod "4cf78616-ccd3-468d-aa28-bf3f4d98e66c" (UID: "4cf78616-ccd3-468d-aa28-bf3f4d98e66c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:33.123065 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.122718 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:33.123065 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.122749 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:33.124579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.124549 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-xks-ca" pod "4cf78616-ccd3-468d-aa28-bf3f4d98e66c" (UID: "4cf78616-ccd3-468d-aa28-bf3f4d98e66c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:33.124663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.124591 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-kube-api-access-wdd9p" pod "4cf78616-ccd3-468d-aa28-bf3f4d98e66c" (UID: "4cf78616-ccd3-468d-aa28-bf3f4d98e66c"). InnerVolumeSpecName "kube-api-access-wdd9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:33.124663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.124613 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d43de03-9d03-4b5d-8fee-350088da28e9-kube-api-access-k9j7m" pod "8d43de03-9d03-4b5d-8fee-350088da28e9" (UID: "8d43de03-9d03-4b5d-8fee-350088da28e9"). InnerVolumeSpecName "kube-api-access-k9j7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:33.124663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.124638 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-xks-ca" pod "8d43de03-9d03-4b5d-8fee-350088da28e9" (UID: "8d43de03-9d03-4b5d-8fee-350088da28e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:33.224495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.224448 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:33.224495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.224486 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9j7m\" (UniqueName: \"kubernetes.io/projected/8d43de03-9d03-4b5d-8fee-350088da28e9-kube-api-access-k9j7m\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:33.224495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.224504 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d43de03-9d03-4b5d-8fee-350088da28e9-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:33.224753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.224516 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdd9p\" (UniqueName: \"kubernetes.io/projected/4cf78616-ccd3-468d-aa28-bf3f4d98e66c-kube-api-access-wdd9p\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:33.751209 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.751081 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" Nov 03 20:34:33.753147 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.753122 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" Nov 03 20:34:33.762173 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.762134 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" path="/var/lib/kubelet/pods/5c089cbb-1f1d-4a87-97c1-b77de0012dda/volumes" Nov 03 20:34:33.762867 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.762838 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d37f893-d432-4047-8b6b-141094b1524f" path="/var/lib/kubelet/pods/5d37f893-d432-4047-8b6b-141094b1524f/volumes" Nov 03 20:34:33.764052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.763748 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4" event={"ID":"8d43de03-9d03-4b5d-8fee-350088da28e9","Type":"ContainerDied","Data":"c654085b276047ac12db2046c9a7be5c7f8b1cd38997f30be458f702428d0b3f"} Nov 03 20:34:33.764052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.763801 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c654085b276047ac12db2046c9a7be5c7f8b1cd38997f30be458f702428d0b3f" Nov 03 20:34:33.764052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.763816 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr" event={"ID":"4cf78616-ccd3-468d-aa28-bf3f4d98e66c","Type":"ContainerDied","Data":"c99de8d94c1da028eda07b7b86472b6d7ceeb4abbc32b0aeb269ae16713c0dbb"} Nov 03 20:34:33.764052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:33.763840 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c99de8d94c1da028eda07b7b86472b6d7ceeb4abbc32b0aeb269ae16713c0dbb" Nov 03 20:34:36.153643 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:36.153598 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:34:36.892640 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:36.892601 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp"] Nov 03 20:34:36.916632 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:36.916592 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq"] Nov 03 20:34:37.465412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.465057 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr"] Nov 03 20:34:37.465412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.465314 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" podUID="cf522b75-3374-41c3-8166-00e272080354" containerName="istio-proxy" containerID="cri-o://c81cbbfd8523c5fc065cc3d85c94b77cd21344601460bf8c42e06393bc025cde" gracePeriod=30 Nov 03 20:34:37.973133 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.973091 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst"] Nov 03 20:34:37.974110 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974086 4457 cpu_manager.go:414] "containerMap: removing container" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" Nov 03 20:34:37.974197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974114 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" Nov 03 20:34:37.974197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974146 4457 cpu_manager.go:414] "containerMap: removing container" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" Nov 03 20:34:37.974197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974154 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" Nov 03 20:34:37.974197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974175 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2a201136-0461-40e8-8388-ed9b7475223f" containerName="main" Nov 03 20:34:37.974197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974183 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a201136-0461-40e8-8388-ed9b7475223f" containerName="main" Nov 03 20:34:37.974444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974234 4457 cpu_manager.go:414] "containerMap: removing container" podUID="8d43de03-9d03-4b5d-8fee-350088da28e9" containerName="revoke-keys" Nov 03 20:34:37.974444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974242 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d43de03-9d03-4b5d-8fee-350088da28e9" containerName="revoke-keys" Nov 03 20:34:37.974444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974261 4457 cpu_manager.go:414] "containerMap: removing container" podUID="4cf78616-ccd3-468d-aa28-bf3f4d98e66c" containerName="revoke-keys" Nov 03 20:34:37.974444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974269 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf78616-ccd3-468d-aa28-bf3f4d98e66c" containerName="revoke-keys" Nov 03 20:34:37.974444 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974440 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a201136-0461-40e8-8388-ed9b7475223f" containerName="main" Nov 03 20:34:37.974685 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974460 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d43de03-9d03-4b5d-8fee-350088da28e9" containerName="revoke-keys" Nov 03 20:34:37.974685 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974473 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf78616-ccd3-468d-aa28-bf3f4d98e66c" containerName="revoke-keys" Nov 03 20:34:37.974685 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974489 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d37f893-d432-4047-8b6b-141094b1524f" containerName="main" Nov 03 20:34:37.974685 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.974500 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c089cbb-1f1d-4a87-97c1-b77de0012dda" containerName="main" Nov 03 20:34:37.980209 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:37.980184 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" Nov 03 20:34:38.007467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.005414 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst"] Nov 03 20:34:38.043483 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.040503 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:34:38.043483 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.040530 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:34:38.116439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.116381 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" podUID="c0e677dc-c502-43a4-9bdf-d7983a6f9b1b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:34:38.135797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.135764 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst\" (UID: \"f8cce7c2-a80e-4336-aca9-f61c47586e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" Nov 03 20:34:38.135982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.135879 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst\" (UID: \"f8cce7c2-a80e-4336-aca9-f61c47586e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" Nov 03 20:34:38.135982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.135907 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89629\" (UniqueName: \"kubernetes.io/projected/f8cce7c2-a80e-4336-aca9-f61c47586e38-kube-api-access-89629\") pod \"maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst\" (UID: \"f8cce7c2-a80e-4336-aca9-f61c47586e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" Nov 03 20:34:38.237924 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.237831 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst\" (UID: \"f8cce7c2-a80e-4336-aca9-f61c47586e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" Nov 03 20:34:38.237924 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.237896 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst\" (UID: \"f8cce7c2-a80e-4336-aca9-f61c47586e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" Nov 03 20:34:38.238174 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.238065 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89629\" (UniqueName: \"kubernetes.io/projected/f8cce7c2-a80e-4336-aca9-f61c47586e38-kube-api-access-89629\") pod \"maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst\" (UID: \"f8cce7c2-a80e-4336-aca9-f61c47586e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" Nov 03 20:34:38.238653 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.238625 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst\" (UID: \"f8cce7c2-a80e-4336-aca9-f61c47586e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" Nov 03 20:34:38.240310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.240286 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst\" (UID: \"f8cce7c2-a80e-4336-aca9-f61c47586e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" Nov 03 20:34:38.250557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.250523 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89629\" (UniqueName: \"kubernetes.io/projected/f8cce7c2-a80e-4336-aca9-f61c47586e38-kube-api-access-89629\") pod \"maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst\" (UID: \"f8cce7c2-a80e-4336-aca9-f61c47586e38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" Nov 03 20:34:38.307577 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.307537 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" Nov 03 20:34:38.600803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.600772 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst"] Nov 03 20:34:38.602839 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:34:38.602812 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cce7c2_a80e_4336_aca9_f61c47586e38.slice/crio-361b673302ed84c6b37680a8eb9c8a5e6b965d3e7cba577b0885f1fbab495b5e WatchSource:0}: Error finding container 361b673302ed84c6b37680a8eb9c8a5e6b965d3e7cba577b0885f1fbab495b5e: Status 404 returned error can't find the container with id 361b673302ed84c6b37680a8eb9c8a5e6b965d3e7cba577b0885f1fbab495b5e Nov 03 20:34:38.810783 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:38.810750 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" event={"ID":"f8cce7c2-a80e-4336-aca9-f61c47586e38","Type":"ContainerStarted","Data":"361b673302ed84c6b37680a8eb9c8a5e6b965d3e7cba577b0885f1fbab495b5e"} Nov 03 20:34:39.029606 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:39.029570 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" podUID="a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:34:39.133691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:39.133603 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" podUID="896c36e5-f4a0-479e-b3c9-7f16881b272e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:34:39.845277 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:39.844954 4457 generic.go:356] "Generic (PLEG): container finished" podID="f8cce7c2-a80e-4336-aca9-f61c47586e38" containerID="62d0cd2f957ecc2251128e5a2a3694526f10ec73146b7c5984c3acf3396c1ca3" exitCode=0 Nov 03 20:34:39.845277 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:39.845220 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" event={"ID":"f8cce7c2-a80e-4336-aca9-f61c47586e38","Type":"ContainerDied","Data":"62d0cd2f957ecc2251128e5a2a3694526f10ec73146b7c5984c3acf3396c1ca3"} Nov 03 20:34:41.028973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.028946 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" Nov 03 20:34:41.182090 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.181998 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-xks-ca\") pod \"f8cce7c2-a80e-4336-aca9-f61c47586e38\" (UID: \"f8cce7c2-a80e-4336-aca9-f61c47586e38\") " Nov 03 20:34:41.182271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.182101 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-openshift-ca\") pod \"f8cce7c2-a80e-4336-aca9-f61c47586e38\" (UID: \"f8cce7c2-a80e-4336-aca9-f61c47586e38\") " Nov 03 20:34:41.182271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.182150 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8cce7c2-a80e-4336-aca9-f61c47586e38-kube-api-access-89629\" (UniqueName: \"kubernetes.io/projected/f8cce7c2-a80e-4336-aca9-f61c47586e38-kube-api-access-89629\") pod \"f8cce7c2-a80e-4336-aca9-f61c47586e38\" (UID: \"f8cce7c2-a80e-4336-aca9-f61c47586e38\") " Nov 03 20:34:41.182946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.182545 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-openshift-ca" pod "f8cce7c2-a80e-4336-aca9-f61c47586e38" (UID: "f8cce7c2-a80e-4336-aca9-f61c47586e38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:41.182946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.182941 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:41.184924 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.184896 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-xks-ca" pod "f8cce7c2-a80e-4336-aca9-f61c47586e38" (UID: "f8cce7c2-a80e-4336-aca9-f61c47586e38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:41.186006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.185978 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cce7c2-a80e-4336-aca9-f61c47586e38-kube-api-access-89629" pod "f8cce7c2-a80e-4336-aca9-f61c47586e38" (UID: "f8cce7c2-a80e-4336-aca9-f61c47586e38"). InnerVolumeSpecName "kube-api-access-89629". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:41.284190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.284153 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89629\" (UniqueName: \"kubernetes.io/projected/f8cce7c2-a80e-4336-aca9-f61c47586e38-kube-api-access-89629\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:41.284190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.284188 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8cce7c2-a80e-4336-aca9-f61c47586e38-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:41.421639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.421601 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q"] Nov 03 20:34:41.867238 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.867195 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" Nov 03 20:34:41.867238 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.867220 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst" event={"ID":"f8cce7c2-a80e-4336-aca9-f61c47586e38","Type":"ContainerDied","Data":"361b673302ed84c6b37680a8eb9c8a5e6b965d3e7cba577b0885f1fbab495b5e"} Nov 03 20:34:41.867497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:41.867281 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="361b673302ed84c6b37680a8eb9c8a5e6b965d3e7cba577b0885f1fbab495b5e" Nov 03 20:34:42.709257 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.709233 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:34:42.763084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.763044 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-certs\") pod \"cf522b75-3374-41c3-8166-00e272080354\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " Nov 03 20:34:42.763285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.763187 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-data\") pod \"cf522b75-3374-41c3-8166-00e272080354\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " Nov 03 20:34:42.763285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.763249 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf522b75-3374-41c3-8166-00e272080354-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf522b75-3374-41c3-8166-00e272080354-istiod-ca-cert\") pod \"cf522b75-3374-41c3-8166-00e272080354\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " Nov 03 20:34:42.763285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.763277 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-socket\") pod \"cf522b75-3374-41c3-8166-00e272080354\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " Nov 03 20:34:42.763559 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.763311 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-envoy\") pod \"cf522b75-3374-41c3-8166-00e272080354\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " Nov 03 20:34:42.763559 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.763352 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-credential-socket\") pod \"cf522b75-3374-41c3-8166-00e272080354\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " Nov 03 20:34:42.763559 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.763436 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf522b75-3374-41c3-8166-00e272080354-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf522b75-3374-41c3-8166-00e272080354-istio-podinfo\") pod \"cf522b75-3374-41c3-8166-00e272080354\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " Nov 03 20:34:42.763559 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.763466 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-istio-token\" (UniqueName: \"kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-istio-token\") pod \"cf522b75-3374-41c3-8166-00e272080354\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " Nov 03 20:34:42.763785 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.763614 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-kube-api-access-zqxzk\" (UniqueName: \"kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-kube-api-access-zqxzk\") pod \"cf522b75-3374-41c3-8166-00e272080354\" (UID: \"cf522b75-3374-41c3-8166-00e272080354\") " Nov 03 20:34:42.764808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.764734 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-socket" pod "cf522b75-3374-41c3-8166-00e272080354" (UID: "cf522b75-3374-41c3-8166-00e272080354"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:42.764937 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.764924 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-certs" pod "cf522b75-3374-41c3-8166-00e272080354" (UID: "cf522b75-3374-41c3-8166-00e272080354"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:42.765098 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.765080 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-data" pod "cf522b75-3374-41c3-8166-00e272080354" (UID: "cf522b75-3374-41c3-8166-00e272080354"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:42.765359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.765340 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf522b75-3374-41c3-8166-00e272080354-istiod-ca-cert" pod "cf522b75-3374-41c3-8166-00e272080354" (UID: "cf522b75-3374-41c3-8166-00e272080354"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:42.765594 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.765573 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-credential-socket" pod "cf522b75-3374-41c3-8166-00e272080354" (UID: "cf522b75-3374-41c3-8166-00e272080354"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:42.767594 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.767556 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-kube-api-access-zqxzk" pod "cf522b75-3374-41c3-8166-00e272080354" (UID: "cf522b75-3374-41c3-8166-00e272080354"). InnerVolumeSpecName "kube-api-access-zqxzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:42.769331 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.769293 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf522b75-3374-41c3-8166-00e272080354-istio-podinfo" pod "cf522b75-3374-41c3-8166-00e272080354" (UID: "cf522b75-3374-41c3-8166-00e272080354"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:42.770594 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.770567 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-envoy" pod "cf522b75-3374-41c3-8166-00e272080354" (UID: "cf522b75-3374-41c3-8166-00e272080354"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:42.770705 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.770630 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-istio-token" pod "cf522b75-3374-41c3-8166-00e272080354" (UID: "cf522b75-3374-41c3-8166-00e272080354"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:42.865762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.865727 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.865762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.865762 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf522b75-3374-41c3-8166-00e272080354-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.865982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.865778 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.865982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.865793 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqxzk\" (UniqueName: \"kubernetes.io/projected/cf522b75-3374-41c3-8166-00e272080354-kube-api-access-zqxzk\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.865982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.865809 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.865982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.865831 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.865982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.865845 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf522b75-3374-41c3-8166-00e272080354-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.865982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.865858 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.865982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.865872 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf522b75-3374-41c3-8166-00e272080354-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:42.879760 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.879152 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m"] Nov 03 20:34:42.879760 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.879382 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" containerID="cri-o://eefdb561dfcd553f17780b48e96c91fc94ba17d822776fc152ff06808d24f714" gracePeriod=30 Nov 03 20:34:42.894799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.894512 4457 generic.go:356] "Generic (PLEG): container finished" podID="cf522b75-3374-41c3-8166-00e272080354" containerID="c81cbbfd8523c5fc065cc3d85c94b77cd21344601460bf8c42e06393bc025cde" exitCode=0 Nov 03 20:34:42.894799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.894556 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" Nov 03 20:34:42.894799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.894575 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" event={"ID":"cf522b75-3374-41c3-8166-00e272080354","Type":"ContainerDied","Data":"c81cbbfd8523c5fc065cc3d85c94b77cd21344601460bf8c42e06393bc025cde"} Nov 03 20:34:42.894799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.894628 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr" event={"ID":"cf522b75-3374-41c3-8166-00e272080354","Type":"ContainerDied","Data":"de0b0a633e8a5a661ede4642c634df525c373418e140c5bdadedc7744e8803d0"} Nov 03 20:34:42.894799 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.894657 4457 scope.go:122] "RemoveContainer" containerID="c81cbbfd8523c5fc065cc3d85c94b77cd21344601460bf8c42e06393bc025cde" Nov 03 20:34:42.913645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.913617 4457 scope.go:122] "RemoveContainer" containerID="c81cbbfd8523c5fc065cc3d85c94b77cd21344601460bf8c42e06393bc025cde" Nov 03 20:34:42.914054 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:42.914030 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81cbbfd8523c5fc065cc3d85c94b77cd21344601460bf8c42e06393bc025cde\": container with ID starting with c81cbbfd8523c5fc065cc3d85c94b77cd21344601460bf8c42e06393bc025cde not found: ID does not exist" containerID="c81cbbfd8523c5fc065cc3d85c94b77cd21344601460bf8c42e06393bc025cde" Nov 03 20:34:42.914182 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.914063 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81cbbfd8523c5fc065cc3d85c94b77cd21344601460bf8c42e06393bc025cde"} err="failed to get container status \"c81cbbfd8523c5fc065cc3d85c94b77cd21344601460bf8c42e06393bc025cde\": rpc error: code = NotFound desc = could not find container \"c81cbbfd8523c5fc065cc3d85c94b77cd21344601460bf8c42e06393bc025cde\": container with ID starting with c81cbbfd8523c5fc065cc3d85c94b77cd21344601460bf8c42e06393bc025cde not found: ID does not exist" Nov 03 20:34:42.927094 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.927053 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr"] Nov 03 20:34:42.932263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:42.932239 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c2c601-openshift-default-6db69b44df-htxnr"] Nov 03 20:34:43.759719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:43.759679 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf522b75-3374-41c3-8166-00e272080354" path="/var/lib/kubelet/pods/cf522b75-3374-41c3-8166-00e272080354/volumes" Nov 03 20:34:44.747774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.747738 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx"] Nov 03 20:34:44.748335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.748319 4457 cpu_manager.go:414] "containerMap: removing container" podUID="f8cce7c2-a80e-4336-aca9-f61c47586e38" containerName="revoke-keys" Nov 03 20:34:44.748424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.748336 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8cce7c2-a80e-4336-aca9-f61c47586e38" containerName="revoke-keys" Nov 03 20:34:44.748424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.748377 4457 cpu_manager.go:414] "containerMap: removing container" podUID="cf522b75-3374-41c3-8166-00e272080354" containerName="istio-proxy" Nov 03 20:34:44.748424 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.748395 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf522b75-3374-41c3-8166-00e272080354" containerName="istio-proxy" Nov 03 20:34:44.748528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.748471 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8cce7c2-a80e-4336-aca9-f61c47586e38" containerName="revoke-keys" Nov 03 20:34:44.748528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.748490 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf522b75-3374-41c3-8166-00e272080354" containerName="istio-proxy" Nov 03 20:34:44.761554 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.761521 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" Nov 03 20:34:44.775998 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.770135 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:34:44.776502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.776190 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx"] Nov 03 20:34:44.777906 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.777879 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:34:44.891117 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.891079 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6833322-n22mx\" (UID: \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" Nov 03 20:34:44.891310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.891261 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6833322-n22mx\" (UID: \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" Nov 03 20:34:44.891428 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.891402 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt565\" (UniqueName: \"kubernetes.io/projected/dd7bc05e-97c6-4904-9dec-aa88a2beef33-kube-api-access-mt565\") pod \"maas-api-revoke-sub-e6833322-n22mx\" (UID: \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" Nov 03 20:34:44.992536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.992499 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt565\" (UniqueName: \"kubernetes.io/projected/dd7bc05e-97c6-4904-9dec-aa88a2beef33-kube-api-access-mt565\") pod \"maas-api-revoke-sub-e6833322-n22mx\" (UID: \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" Nov 03 20:34:44.992745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.992701 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6833322-n22mx\" (UID: \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" Nov 03 20:34:44.992884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.992861 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6833322-n22mx\" (UID: \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" Nov 03 20:34:44.993725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.993696 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6833322-n22mx\" (UID: \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" Nov 03 20:34:44.995664 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:44.995636 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6833322-n22mx\" (UID: \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" Nov 03 20:34:45.006866 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:45.006804 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt565\" (UniqueName: \"kubernetes.io/projected/dd7bc05e-97c6-4904-9dec-aa88a2beef33-kube-api-access-mt565\") pod \"maas-api-revoke-sub-e6833322-n22mx\" (UID: \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" Nov 03 20:34:45.085643 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:45.085604 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" Nov 03 20:34:45.555058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:45.555012 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx"] Nov 03 20:34:45.817275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:45.817236 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf"] Nov 03 20:34:45.822377 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:45.822351 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-62221d-hqwrf"] Nov 03 20:34:45.966189 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:45.966139 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" event={"ID":"dd7bc05e-97c6-4904-9dec-aa88a2beef33","Type":"ContainerStarted","Data":"5997256a511bad39e436191aca7d8adf4f7de4c7cbb5cae11c21fc5fdaf03434"} Nov 03 20:34:46.422987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.422944 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" podUID="df7f2048-8b4e-46ef-a5b9-e2504a0be18c" containerName="payload-processing" containerID="cri-o://3e262bb946869e428be51ab90c73d5c5ab96ddb937d4e42881e5824bb0c5f5e9" gracePeriod=25 Nov 03 20:34:46.802660 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.802628 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" Nov 03 20:34:46.807151 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.807125 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" Nov 03 20:34:46.925840 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.925750 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-kube-api-access-4xjwg\" (UniqueName: \"kubernetes.io/projected/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-kube-api-access-4xjwg\") pod \"df7f2048-8b4e-46ef-a5b9-e2504a0be18c\" (UID: \"df7f2048-8b4e-46ef-a5b9-e2504a0be18c\") " Nov 03 20:34:46.926213 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.925874 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-plugins-config-volume\") pod \"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58\" (UID: \"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58\") " Nov 03 20:34:46.926213 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.925960 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-plugins-config-volume\") pod \"df7f2048-8b4e-46ef-a5b9-e2504a0be18c\" (UID: \"df7f2048-8b4e-46ef-a5b9-e2504a0be18c\") " Nov 03 20:34:46.926213 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.925995 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-kube-api-access-k22hp\" (UniqueName: \"kubernetes.io/projected/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-kube-api-access-k22hp\") pod \"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58\" (UID: \"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58\") " Nov 03 20:34:46.926381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.926300 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-plugins-config-volume" pod "a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58" (UID: "a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:46.926476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.926409 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-plugins-config-volume" pod "df7f2048-8b4e-46ef-a5b9-e2504a0be18c" (UID: "df7f2048-8b4e-46ef-a5b9-e2504a0be18c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:46.926673 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.926654 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:46.926723 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.926678 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:46.928009 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.927985 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-kube-api-access-4xjwg" pod "df7f2048-8b4e-46ef-a5b9-e2504a0be18c" (UID: "df7f2048-8b4e-46ef-a5b9-e2504a0be18c"). InnerVolumeSpecName "kube-api-access-4xjwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:46.928146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.928125 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-kube-api-access-k22hp" pod "a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58" (UID: "a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58"). InnerVolumeSpecName "kube-api-access-k22hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:46.976993 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.976955 4457 generic.go:356] "Generic (PLEG): container finished" podID="df7f2048-8b4e-46ef-a5b9-e2504a0be18c" containerID="3e262bb946869e428be51ab90c73d5c5ab96ddb937d4e42881e5824bb0c5f5e9" exitCode=0 Nov 03 20:34:46.977239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.977017 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" Nov 03 20:34:46.977239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.977043 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" event={"ID":"df7f2048-8b4e-46ef-a5b9-e2504a0be18c","Type":"ContainerDied","Data":"3e262bb946869e428be51ab90c73d5c5ab96ddb937d4e42881e5824bb0c5f5e9"} Nov 03 20:34:46.977239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.977101 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q" event={"ID":"df7f2048-8b4e-46ef-a5b9-e2504a0be18c","Type":"ContainerDied","Data":"b6e93d570f504aae13a190dbb8269b0b2fcfbf9724630fa9852e9162eae4f8ca"} Nov 03 20:34:46.977239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.977116 4457 scope.go:122] "RemoveContainer" containerID="3e262bb946869e428be51ab90c73d5c5ab96ddb937d4e42881e5824bb0c5f5e9" Nov 03 20:34:46.978832 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.978806 4457 generic.go:356] "Generic (PLEG): container finished" podID="a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58" containerID="93891d4861cb8f25d605df77b2b90208dbcd7462045025c23dc7224085eb9c23" exitCode=0 Nov 03 20:34:46.978945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.978855 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" Nov 03 20:34:46.978945 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.978892 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" event={"ID":"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58","Type":"ContainerDied","Data":"93891d4861cb8f25d605df77b2b90208dbcd7462045025c23dc7224085eb9c23"} Nov 03 20:34:46.979050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.978950 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t" event={"ID":"a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58","Type":"ContainerDied","Data":"1c19e4d6f2e951b493142cb129506c74adc80c46d523144cf95758c427ecde7c"} Nov 03 20:34:46.984661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.984633 4457 generic.go:356] "Generic (PLEG): container finished" podID="dd7bc05e-97c6-4904-9dec-aa88a2beef33" containerID="25b9514eb6e2ee6c474620e9ae8a3c7b0cccd71a896da13c405394da2ebe1000" exitCode=0 Nov 03 20:34:46.984800 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.984780 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" event={"ID":"dd7bc05e-97c6-4904-9dec-aa88a2beef33","Type":"ContainerDied","Data":"25b9514eb6e2ee6c474620e9ae8a3c7b0cccd71a896da13c405394da2ebe1000"} Nov 03 20:34:46.990576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.990555 4457 scope.go:122] "RemoveContainer" containerID="3e262bb946869e428be51ab90c73d5c5ab96ddb937d4e42881e5824bb0c5f5e9" Nov 03 20:34:46.990908 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:46.990874 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e262bb946869e428be51ab90c73d5c5ab96ddb937d4e42881e5824bb0c5f5e9\": container with ID starting with 3e262bb946869e428be51ab90c73d5c5ab96ddb937d4e42881e5824bb0c5f5e9 not found: ID does not exist" containerID="3e262bb946869e428be51ab90c73d5c5ab96ddb937d4e42881e5824bb0c5f5e9" Nov 03 20:34:46.991000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.990917 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e262bb946869e428be51ab90c73d5c5ab96ddb937d4e42881e5824bb0c5f5e9"} err="failed to get container status \"3e262bb946869e428be51ab90c73d5c5ab96ddb937d4e42881e5824bb0c5f5e9\": rpc error: code = NotFound desc = could not find container \"3e262bb946869e428be51ab90c73d5c5ab96ddb937d4e42881e5824bb0c5f5e9\": container with ID starting with 3e262bb946869e428be51ab90c73d5c5ab96ddb937d4e42881e5824bb0c5f5e9 not found: ID does not exist" Nov 03 20:34:46.991000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:46.990941 4457 scope.go:122] "RemoveContainer" containerID="93891d4861cb8f25d605df77b2b90208dbcd7462045025c23dc7224085eb9c23" Nov 03 20:34:47.047527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:47.030544 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xjwg\" (UniqueName: \"kubernetes.io/projected/df7f2048-8b4e-46ef-a5b9-e2504a0be18c-kube-api-access-4xjwg\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:47.047527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:47.030729 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k22hp\" (UniqueName: \"kubernetes.io/projected/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58-kube-api-access-k22hp\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:47.047527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:47.036407 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t"] Nov 03 20:34:47.047527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:47.044915 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-62221d-f79997dbb-vjc6t"] Nov 03 20:34:47.056502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:47.051750 4457 scope.go:122] "RemoveContainer" containerID="93891d4861cb8f25d605df77b2b90208dbcd7462045025c23dc7224085eb9c23" Nov 03 20:34:47.056502 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:47.053328 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93891d4861cb8f25d605df77b2b90208dbcd7462045025c23dc7224085eb9c23\": container with ID starting with 93891d4861cb8f25d605df77b2b90208dbcd7462045025c23dc7224085eb9c23 not found: ID does not exist" containerID="93891d4861cb8f25d605df77b2b90208dbcd7462045025c23dc7224085eb9c23" Nov 03 20:34:47.056502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:47.053361 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93891d4861cb8f25d605df77b2b90208dbcd7462045025c23dc7224085eb9c23"} err="failed to get container status \"93891d4861cb8f25d605df77b2b90208dbcd7462045025c23dc7224085eb9c23\": rpc error: code = NotFound desc = could not find container \"93891d4861cb8f25d605df77b2b90208dbcd7462045025c23dc7224085eb9c23\": container with ID starting with 93891d4861cb8f25d605df77b2b90208dbcd7462045025c23dc7224085eb9c23 not found: ID does not exist" Nov 03 20:34:47.067439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:47.066133 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q"] Nov 03 20:34:47.070498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:47.070464 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-1c6a98-8cfdccd74-c5w6q"] Nov 03 20:34:47.759719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:47.759675 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83cbc1d7-7635-4251-9b60-1c5a0b4bcf24" path="/var/lib/kubelet/pods/83cbc1d7-7635-4251-9b60-1c5a0b4bcf24/volumes" Nov 03 20:34:47.760307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:47.760288 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58" path="/var/lib/kubelet/pods/a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58/volumes" Nov 03 20:34:47.760830 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:47.760813 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df7f2048-8b4e-46ef-a5b9-e2504a0be18c" path="/var/lib/kubelet/pods/df7f2048-8b4e-46ef-a5b9-e2504a0be18c/volumes" Nov 03 20:34:48.111680 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.111299 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj"] Nov 03 20:34:48.115773 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.115740 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" podUID="c0e677dc-c502-43a4-9bdf-d7983a6f9b1b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:34:48.217601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.217568 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" Nov 03 20:34:48.310475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.310448 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:34:48.380409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.377360 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-xks-ca\") pod \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\" (UID: \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\") " Nov 03 20:34:48.397420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.384796 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-openshift-ca" pod "dd7bc05e-97c6-4904-9dec-aa88a2beef33" (UID: "dd7bc05e-97c6-4904-9dec-aa88a2beef33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:48.397420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.390564 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-openshift-ca\") pod \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\" (UID: \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\") " Nov 03 20:34:48.397420 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.390730 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd7bc05e-97c6-4904-9dec-aa88a2beef33-kube-api-access-mt565\" (UniqueName: \"kubernetes.io/projected/dd7bc05e-97c6-4904-9dec-aa88a2beef33-kube-api-access-mt565\") pod \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\" (UID: \"dd7bc05e-97c6-4904-9dec-aa88a2beef33\") " Nov 03 20:34:48.422414 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.409293 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:48.436445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.431465 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-xks-ca" pod "dd7bc05e-97c6-4904-9dec-aa88a2beef33" (UID: "dd7bc05e-97c6-4904-9dec-aa88a2beef33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:48.436445 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.435326 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7bc05e-97c6-4904-9dec-aa88a2beef33-kube-api-access-mt565" pod "dd7bc05e-97c6-4904-9dec-aa88a2beef33" (UID: "dd7bc05e-97c6-4904-9dec-aa88a2beef33"). InnerVolumeSpecName "kube-api-access-mt565". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:48.509927 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.509895 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-podinfo\") pod \"18c6aa2c-4423-4872-85e3-33983cb81c2b\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " Nov 03 20:34:48.510111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.509939 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-token\" (UniqueName: \"kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-token\") pod \"18c6aa2c-4423-4872-85e3-33983cb81c2b\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " Nov 03 20:34:48.510111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.509977 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-data\") pod \"18c6aa2c-4423-4872-85e3-33983cb81c2b\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " Nov 03 20:34:48.510111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.510009 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-envoy\") pod \"18c6aa2c-4423-4872-85e3-33983cb81c2b\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " Nov 03 20:34:48.510111 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.510095 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-credential-socket\") pod \"18c6aa2c-4423-4872-85e3-33983cb81c2b\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " Nov 03 20:34:48.510402 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.510129 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18c6aa2c-4423-4872-85e3-33983cb81c2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c6aa2c-4423-4872-85e3-33983cb81c2b-istiod-ca-cert\") pod \"18c6aa2c-4423-4872-85e3-33983cb81c2b\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " Nov 03 20:34:48.510402 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.510190 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-certs\") pod \"18c6aa2c-4423-4872-85e3-33983cb81c2b\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " Nov 03 20:34:48.510402 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.510245 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-socket\") pod \"18c6aa2c-4423-4872-85e3-33983cb81c2b\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " Nov 03 20:34:48.510402 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.510296 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-data" pod "18c6aa2c-4423-4872-85e3-33983cb81c2b" (UID: "18c6aa2c-4423-4872-85e3-33983cb81c2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:48.510402 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.510319 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-kube-api-access-l7qd7\" (UniqueName: \"kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-kube-api-access-l7qd7\") pod \"18c6aa2c-4423-4872-85e3-33983cb81c2b\" (UID: \"18c6aa2c-4423-4872-85e3-33983cb81c2b\") " Nov 03 20:34:48.510681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.510526 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-certs" pod "18c6aa2c-4423-4872-85e3-33983cb81c2b" (UID: "18c6aa2c-4423-4872-85e3-33983cb81c2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:48.510748 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.510724 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-credential-socket" pod "18c6aa2c-4423-4872-85e3-33983cb81c2b" (UID: "18c6aa2c-4423-4872-85e3-33983cb81c2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:48.510864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.510831 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18c6aa2c-4423-4872-85e3-33983cb81c2b-istiod-ca-cert" pod "18c6aa2c-4423-4872-85e3-33983cb81c2b" (UID: "18c6aa2c-4423-4872-85e3-33983cb81c2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:48.510925 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.510669 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-socket" pod "18c6aa2c-4423-4872-85e3-33983cb81c2b" (UID: "18c6aa2c-4423-4872-85e3-33983cb81c2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:48.511035 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.511010 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:48.511169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.511039 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:48.511169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.511060 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18c6aa2c-4423-4872-85e3-33983cb81c2b-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:48.511169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.511078 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt565\" (UniqueName: \"kubernetes.io/projected/dd7bc05e-97c6-4904-9dec-aa88a2beef33-kube-api-access-mt565\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:48.511169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.511093 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:48.511169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.511106 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:48.511169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.511120 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd7bc05e-97c6-4904-9dec-aa88a2beef33-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:48.512440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.512415 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-envoy" pod "18c6aa2c-4423-4872-85e3-33983cb81c2b" (UID: "18c6aa2c-4423-4872-85e3-33983cb81c2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:48.512791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.512767 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-kube-api-access-l7qd7" pod "18c6aa2c-4423-4872-85e3-33983cb81c2b" (UID: "18c6aa2c-4423-4872-85e3-33983cb81c2b"). InnerVolumeSpecName "kube-api-access-l7qd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:48.513070 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.513056 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-token" pod "18c6aa2c-4423-4872-85e3-33983cb81c2b" (UID: "18c6aa2c-4423-4872-85e3-33983cb81c2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:48.513305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.513293 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-podinfo" pod "18c6aa2c-4423-4872-85e3-33983cb81c2b" (UID: "18c6aa2c-4423-4872-85e3-33983cb81c2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:48.611812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.611775 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:48.611812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.611806 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7qd7\" (UniqueName: \"kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-kube-api-access-l7qd7\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:48.611812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.611817 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:48.612107 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:48.611826 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18c6aa2c-4423-4872-85e3-33983cb81c2b-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:49.021264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.021172 4457 generic.go:356] "Generic (PLEG): container finished" podID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerID="eefdb561dfcd553f17780b48e96c91fc94ba17d822776fc152ff06808d24f714" exitCode=0 Nov 03 20:34:49.021264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.021219 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" Nov 03 20:34:49.021264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.021252 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" event={"ID":"18c6aa2c-4423-4872-85e3-33983cb81c2b","Type":"ContainerDied","Data":"eefdb561dfcd553f17780b48e96c91fc94ba17d822776fc152ff06808d24f714"} Nov 03 20:34:49.021555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.021314 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m" event={"ID":"18c6aa2c-4423-4872-85e3-33983cb81c2b","Type":"ContainerDied","Data":"e0f9364ff310395620efadedd52b959228235172fbb35e12c1adb46193b10564"} Nov 03 20:34:49.021555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.021344 4457 scope.go:122] "RemoveContainer" containerID="eefdb561dfcd553f17780b48e96c91fc94ba17d822776fc152ff06808d24f714" Nov 03 20:34:49.023481 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.023425 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" event={"ID":"dd7bc05e-97c6-4904-9dec-aa88a2beef33","Type":"ContainerDied","Data":"5997256a511bad39e436191aca7d8adf4f7de4c7cbb5cae11c21fc5fdaf03434"} Nov 03 20:34:49.023584 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.023479 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx" Nov 03 20:34:49.023584 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.023482 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5997256a511bad39e436191aca7d8adf4f7de4c7cbb5cae11c21fc5fdaf03434" Nov 03 20:34:49.033736 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.033713 4457 scope.go:122] "RemoveContainer" containerID="eefdb561dfcd553f17780b48e96c91fc94ba17d822776fc152ff06808d24f714" Nov 03 20:34:49.034068 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:49.034048 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eefdb561dfcd553f17780b48e96c91fc94ba17d822776fc152ff06808d24f714\": container with ID starting with eefdb561dfcd553f17780b48e96c91fc94ba17d822776fc152ff06808d24f714 not found: ID does not exist" containerID="eefdb561dfcd553f17780b48e96c91fc94ba17d822776fc152ff06808d24f714" Nov 03 20:34:49.034130 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.034077 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eefdb561dfcd553f17780b48e96c91fc94ba17d822776fc152ff06808d24f714"} err="failed to get container status \"eefdb561dfcd553f17780b48e96c91fc94ba17d822776fc152ff06808d24f714\": rpc error: code = NotFound desc = could not find container \"eefdb561dfcd553f17780b48e96c91fc94ba17d822776fc152ff06808d24f714\": container with ID starting with eefdb561dfcd553f17780b48e96c91fc94ba17d822776fc152ff06808d24f714 not found: ID does not exist" Nov 03 20:34:49.048479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.048455 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m"] Nov 03 20:34:49.054656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.054635 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-5bz7m"] Nov 03 20:34:49.134051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.134009 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" podUID="896c36e5-f4a0-479e-b3c9-7f16881b272e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:34:49.216904 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.216870 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps"] Nov 03 20:34:49.217535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.217514 4457 cpu_manager.go:414] "containerMap: removing container" podUID="df7f2048-8b4e-46ef-a5b9-e2504a0be18c" containerName="payload-processing" Nov 03 20:34:49.217535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.217532 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7f2048-8b4e-46ef-a5b9-e2504a0be18c" containerName="payload-processing" Nov 03 20:34:49.217729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.217548 4457 cpu_manager.go:414] "containerMap: removing container" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" Nov 03 20:34:49.217729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.217554 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" Nov 03 20:34:49.217729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.217560 4457 cpu_manager.go:414] "containerMap: removing container" podUID="a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58" containerName="payload-processing" Nov 03 20:34:49.217729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.217566 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58" containerName="payload-processing" Nov 03 20:34:49.217729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.217598 4457 cpu_manager.go:414] "containerMap: removing container" podUID="dd7bc05e-97c6-4904-9dec-aa88a2beef33" containerName="revoke-keys" Nov 03 20:34:49.217729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.217603 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd7bc05e-97c6-4904-9dec-aa88a2beef33" containerName="revoke-keys" Nov 03 20:34:49.217729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.217675 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd7bc05e-97c6-4904-9dec-aa88a2beef33" containerName="revoke-keys" Nov 03 20:34:49.217729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.217686 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7f2048-8b4e-46ef-a5b9-e2504a0be18c" containerName="payload-processing" Nov 03 20:34:49.217729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.217693 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" containerName="istio-proxy" Nov 03 20:34:49.217729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.217698 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c7cba4-4279-4f48-9ecc-e48e6a0e2e58" containerName="payload-processing" Nov 03 20:34:49.223569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.223546 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" Nov 03 20:34:49.231897 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.231867 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps"] Nov 03 20:34:49.314655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.314614 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:34:49.314839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.314796 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:34:49.320785 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.320754 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m497g\" (UniqueName: \"kubernetes.io/projected/d0ca0910-85fc-4163-a801-3f58ea5578e1-kube-api-access-m497g\") pod \"maas-api-revoke-sub-838f0162-2g6ps\" (UID: \"d0ca0910-85fc-4163-a801-3f58ea5578e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" Nov 03 20:34:49.320967 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.320949 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-838f0162-2g6ps\" (UID: \"d0ca0910-85fc-4163-a801-3f58ea5578e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" Nov 03 20:34:49.321033 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.320979 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-838f0162-2g6ps\" (UID: \"d0ca0910-85fc-4163-a801-3f58ea5578e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" Nov 03 20:34:49.422070 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.422033 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-838f0162-2g6ps\" (UID: \"d0ca0910-85fc-4163-a801-3f58ea5578e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" Nov 03 20:34:49.422070 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.422070 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-838f0162-2g6ps\" (UID: \"d0ca0910-85fc-4163-a801-3f58ea5578e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" Nov 03 20:34:49.422323 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.422163 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m497g\" (UniqueName: \"kubernetes.io/projected/d0ca0910-85fc-4163-a801-3f58ea5578e1-kube-api-access-m497g\") pod \"maas-api-revoke-sub-838f0162-2g6ps\" (UID: \"d0ca0910-85fc-4163-a801-3f58ea5578e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" Nov 03 20:34:49.422813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.422788 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-838f0162-2g6ps\" (UID: \"d0ca0910-85fc-4163-a801-3f58ea5578e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" Nov 03 20:34:49.424609 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.424586 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-838f0162-2g6ps\" (UID: \"d0ca0910-85fc-4163-a801-3f58ea5578e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" Nov 03 20:34:49.462934 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.462892 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m497g\" (UniqueName: \"kubernetes.io/projected/d0ca0910-85fc-4163-a801-3f58ea5578e1-kube-api-access-m497g\") pod \"maas-api-revoke-sub-838f0162-2g6ps\" (UID: \"d0ca0910-85fc-4163-a801-3f58ea5578e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" Nov 03 20:34:49.549339 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.549299 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" Nov 03 20:34:49.771084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.771048 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18c6aa2c-4423-4872-85e3-33983cb81c2b" path="/var/lib/kubelet/pods/18c6aa2c-4423-4872-85e3-33983cb81c2b/volumes" Nov 03 20:34:49.823112 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.823079 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" Nov 03 20:34:49.841572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.841538 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps"] Nov 03 20:34:49.894728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.894702 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" Nov 03 20:34:49.928884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.928835 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-plugins-config-volume\") pod \"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b\" (UID: \"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b\") " Nov 03 20:34:49.929067 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.928955 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-kube-api-access-zsbqz\" (UniqueName: \"kubernetes.io/projected/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-kube-api-access-zsbqz\") pod \"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b\" (UID: \"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b\") " Nov 03 20:34:49.929067 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.929053 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896c36e5-f4a0-479e-b3c9-7f16881b272e-kube-api-access-n4987\" (UniqueName: \"kubernetes.io/projected/896c36e5-f4a0-479e-b3c9-7f16881b272e-kube-api-access-n4987\") pod \"896c36e5-f4a0-479e-b3c9-7f16881b272e\" (UID: \"896c36e5-f4a0-479e-b3c9-7f16881b272e\") " Nov 03 20:34:49.929216 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.929120 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896c36e5-f4a0-479e-b3c9-7f16881b272e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896c36e5-f4a0-479e-b3c9-7f16881b272e-plugins-config-volume\") pod \"896c36e5-f4a0-479e-b3c9-7f16881b272e\" (UID: \"896c36e5-f4a0-479e-b3c9-7f16881b272e\") " Nov 03 20:34:49.929286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.929237 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-plugins-config-volume" pod "c0e677dc-c502-43a4-9bdf-d7983a6f9b1b" (UID: "c0e677dc-c502-43a4-9bdf-d7983a6f9b1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:49.929674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.929643 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:49.930021 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.929994 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896c36e5-f4a0-479e-b3c9-7f16881b272e-plugins-config-volume" pod "896c36e5-f4a0-479e-b3c9-7f16881b272e" (UID: "896c36e5-f4a0-479e-b3c9-7f16881b272e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:49.931649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.931627 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-kube-api-access-zsbqz" pod "c0e677dc-c502-43a4-9bdf-d7983a6f9b1b" (UID: "c0e677dc-c502-43a4-9bdf-d7983a6f9b1b"). InnerVolumeSpecName "kube-api-access-zsbqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:49.931746 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:49.931656 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896c36e5-f4a0-479e-b3c9-7f16881b272e-kube-api-access-n4987" pod "896c36e5-f4a0-479e-b3c9-7f16881b272e" (UID: "896c36e5-f4a0-479e-b3c9-7f16881b272e"). InnerVolumeSpecName "kube-api-access-n4987". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:50.030529 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.030498 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4987\" (UniqueName: \"kubernetes.io/projected/896c36e5-f4a0-479e-b3c9-7f16881b272e-kube-api-access-n4987\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:50.030529 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.030525 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/896c36e5-f4a0-479e-b3c9-7f16881b272e-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:50.030529 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.030535 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsbqz\" (UniqueName: \"kubernetes.io/projected/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b-kube-api-access-zsbqz\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:50.034785 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.034751 4457 generic.go:356] "Generic (PLEG): container finished" podID="c0e677dc-c502-43a4-9bdf-d7983a6f9b1b" containerID="eb792ba6fff2a1dc5eca7f82dc1ae3c95d97351e7dcc6e23f3da0898f4bd1818" exitCode=0 Nov 03 20:34:50.034939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.034803 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" Nov 03 20:34:50.034939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.034837 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" event={"ID":"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b","Type":"ContainerDied","Data":"eb792ba6fff2a1dc5eca7f82dc1ae3c95d97351e7dcc6e23f3da0898f4bd1818"} Nov 03 20:34:50.034939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.034921 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv" event={"ID":"c0e677dc-c502-43a4-9bdf-d7983a6f9b1b","Type":"ContainerDied","Data":"a36fa267e1bbcf74c80c82c863c8c179e7c1fd4523f6bc9a23904f76dbfc70d8"} Nov 03 20:34:50.035115 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.034954 4457 scope.go:122] "RemoveContainer" containerID="eb792ba6fff2a1dc5eca7f82dc1ae3c95d97351e7dcc6e23f3da0898f4bd1818" Nov 03 20:34:50.036954 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.036926 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" event={"ID":"d0ca0910-85fc-4163-a801-3f58ea5578e1","Type":"ContainerStarted","Data":"873ef430dda19f0db17cdefd97202350445ed0c40406e8c484fe903fc1452966"} Nov 03 20:34:50.040140 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.040115 4457 generic.go:356] "Generic (PLEG): container finished" podID="896c36e5-f4a0-479e-b3c9-7f16881b272e" containerID="7e7d3c376499619d5e61271a98b04d9a2234399edaf0e6f233ad1b214c5e1255" exitCode=0 Nov 03 20:34:50.040263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.040159 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" Nov 03 20:34:50.040263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.040202 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" event={"ID":"896c36e5-f4a0-479e-b3c9-7f16881b272e","Type":"ContainerDied","Data":"7e7d3c376499619d5e61271a98b04d9a2234399edaf0e6f233ad1b214c5e1255"} Nov 03 20:34:50.040263 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.040248 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g" event={"ID":"896c36e5-f4a0-479e-b3c9-7f16881b272e","Type":"ContainerDied","Data":"31327884ecf7688d79db048aa3b47bb8f8b2588437cde08b0be4002c20f55514"} Nov 03 20:34:50.050141 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.050120 4457 scope.go:122] "RemoveContainer" containerID="eb792ba6fff2a1dc5eca7f82dc1ae3c95d97351e7dcc6e23f3da0898f4bd1818" Nov 03 20:34:50.050482 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:50.050460 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb792ba6fff2a1dc5eca7f82dc1ae3c95d97351e7dcc6e23f3da0898f4bd1818\": container with ID starting with eb792ba6fff2a1dc5eca7f82dc1ae3c95d97351e7dcc6e23f3da0898f4bd1818 not found: ID does not exist" containerID="eb792ba6fff2a1dc5eca7f82dc1ae3c95d97351e7dcc6e23f3da0898f4bd1818" Nov 03 20:34:50.050588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.050493 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb792ba6fff2a1dc5eca7f82dc1ae3c95d97351e7dcc6e23f3da0898f4bd1818"} err="failed to get container status \"eb792ba6fff2a1dc5eca7f82dc1ae3c95d97351e7dcc6e23f3da0898f4bd1818\": rpc error: code = NotFound desc = could not find container \"eb792ba6fff2a1dc5eca7f82dc1ae3c95d97351e7dcc6e23f3da0898f4bd1818\": container with ID starting with eb792ba6fff2a1dc5eca7f82dc1ae3c95d97351e7dcc6e23f3da0898f4bd1818 not found: ID does not exist" Nov 03 20:34:50.050588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.050515 4457 scope.go:122] "RemoveContainer" containerID="7e7d3c376499619d5e61271a98b04d9a2234399edaf0e6f233ad1b214c5e1255" Nov 03 20:34:50.062491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.062446 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv"] Nov 03 20:34:50.065516 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.065494 4457 scope.go:122] "RemoveContainer" containerID="7e7d3c376499619d5e61271a98b04d9a2234399edaf0e6f233ad1b214c5e1255" Nov 03 20:34:50.065874 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:50.065852 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e7d3c376499619d5e61271a98b04d9a2234399edaf0e6f233ad1b214c5e1255\": container with ID starting with 7e7d3c376499619d5e61271a98b04d9a2234399edaf0e6f233ad1b214c5e1255 not found: ID does not exist" containerID="7e7d3c376499619d5e61271a98b04d9a2234399edaf0e6f233ad1b214c5e1255" Nov 03 20:34:50.065961 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.065882 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e7d3c376499619d5e61271a98b04d9a2234399edaf0e6f233ad1b214c5e1255"} err="failed to get container status \"7e7d3c376499619d5e61271a98b04d9a2234399edaf0e6f233ad1b214c5e1255\": rpc error: code = NotFound desc = could not find container \"7e7d3c376499619d5e61271a98b04d9a2234399edaf0e6f233ad1b214c5e1255\": container with ID starting with 7e7d3c376499619d5e61271a98b04d9a2234399edaf0e6f233ad1b214c5e1255 not found: ID does not exist" Nov 03 20:34:50.068027 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.068002 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c2c601-8468fb76c9-htwhv"] Nov 03 20:34:50.083372 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.083349 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g"] Nov 03 20:34:50.091932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:50.091904 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c2c601-5c8fb88646-l9k8g"] Nov 03 20:34:51.054589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:51.054554 4457 generic.go:356] "Generic (PLEG): container finished" podID="d0ca0910-85fc-4163-a801-3f58ea5578e1" containerID="eb13ca2c11e5385d03f1d7d6b3d520739769703b1c671790c220ce01f4918b83" exitCode=0 Nov 03 20:34:51.055072 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:51.054610 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" event={"ID":"d0ca0910-85fc-4163-a801-3f58ea5578e1","Type":"ContainerDied","Data":"eb13ca2c11e5385d03f1d7d6b3d520739769703b1c671790c220ce01f4918b83"} Nov 03 20:34:51.761550 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:51.761517 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896c36e5-f4a0-479e-b3c9-7f16881b272e" path="/var/lib/kubelet/pods/896c36e5-f4a0-479e-b3c9-7f16881b272e/volumes" Nov 03 20:34:51.762071 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:51.762044 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e677dc-c502-43a4-9bdf-d7983a6f9b1b" path="/var/lib/kubelet/pods/c0e677dc-c502-43a4-9bdf-d7983a6f9b1b/volumes" Nov 03 20:34:51.915078 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:51.915034 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" containerID="cri-o://2bfec98d165f28e2a730aa0a33f5a026f669cb0b9cea36b537d87bf37a6208d3" gracePeriod=45 Nov 03 20:34:51.938104 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:51.938064 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" containerID="cri-o://a4863bcc5fe6ab9849a22ebb0926c64f321e44e4d6b2a3949c215f2ef1b85d25" gracePeriod=45 Nov 03 20:34:52.070406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.069302 4457 generic.go:356] "Generic (PLEG): container finished" podID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerID="a4863bcc5fe6ab9849a22ebb0926c64f321e44e4d6b2a3949c215f2ef1b85d25" exitCode=0 Nov 03 20:34:52.070406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.069568 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" event={"ID":"a5a5de13-728b-4a75-8990-807d1d3811e4","Type":"ContainerDied","Data":"a4863bcc5fe6ab9849a22ebb0926c64f321e44e4d6b2a3949c215f2ef1b85d25"} Nov 03 20:34:52.073329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.072489 4457 generic.go:356] "Generic (PLEG): container finished" podID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerID="2bfec98d165f28e2a730aa0a33f5a026f669cb0b9cea36b537d87bf37a6208d3" exitCode=0 Nov 03 20:34:52.073329 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.072557 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" event={"ID":"008491db-6f31-4ea1-9cbf-f9f779af78ea","Type":"ContainerDied","Data":"2bfec98d165f28e2a730aa0a33f5a026f669cb0b9cea36b537d87bf37a6208d3"} Nov 03 20:34:52.249405 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.244445 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:34:52.270409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.261317 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-tmp-dir\") pod \"008491db-6f31-4ea1-9cbf-f9f779af78ea\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " Nov 03 20:34:52.270409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.261369 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-home\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-home\") pod \"008491db-6f31-4ea1-9cbf-f9f779af78ea\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " Nov 03 20:34:52.270409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.261468 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-dshm\") pod \"008491db-6f31-4ea1-9cbf-f9f779af78ea\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " Nov 03 20:34:52.270409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.261518 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008491db-6f31-4ea1-9cbf-f9f779af78ea-kube-api-access-v28h5\" (UniqueName: \"kubernetes.io/projected/008491db-6f31-4ea1-9cbf-f9f779af78ea-kube-api-access-v28h5\") pod \"008491db-6f31-4ea1-9cbf-f9f779af78ea\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " Nov 03 20:34:52.270409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.261649 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-model-cache\") pod \"008491db-6f31-4ea1-9cbf-f9f779af78ea\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " Nov 03 20:34:52.270409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.261729 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008491db-6f31-4ea1-9cbf-f9f779af78ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/008491db-6f31-4ea1-9cbf-f9f779af78ea-tls-certs\") pod \"008491db-6f31-4ea1-9cbf-f9f779af78ea\" (UID: \"008491db-6f31-4ea1-9cbf-f9f779af78ea\") " Nov 03 20:34:52.270409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.263138 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-model-cache" pod "008491db-6f31-4ea1-9cbf-f9f779af78ea" (UID: "008491db-6f31-4ea1-9cbf-f9f779af78ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.270409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.263205 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-tmp-dir" pod "008491db-6f31-4ea1-9cbf-f9f779af78ea" (UID: "008491db-6f31-4ea1-9cbf-f9f779af78ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.270409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.263251 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-home" pod "008491db-6f31-4ea1-9cbf-f9f779af78ea" (UID: "008491db-6f31-4ea1-9cbf-f9f779af78ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.281019 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.279243 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008491db-6f31-4ea1-9cbf-f9f779af78ea-kube-api-access-v28h5" pod "008491db-6f31-4ea1-9cbf-f9f779af78ea" (UID: "008491db-6f31-4ea1-9cbf-f9f779af78ea"). InnerVolumeSpecName "kube-api-access-v28h5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:52.293595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.293562 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008491db-6f31-4ea1-9cbf-f9f779af78ea-tls-certs" pod "008491db-6f31-4ea1-9cbf-f9f779af78ea" (UID: "008491db-6f31-4ea1-9cbf-f9f779af78ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:52.297566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.297538 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-dshm" pod "008491db-6f31-4ea1-9cbf-f9f779af78ea" (UID: "008491db-6f31-4ea1-9cbf-f9f779af78ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.299267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.298907 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" Nov 03 20:34:52.303605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.303583 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:34:52.364310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.364275 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0ca0910-85fc-4163-a801-3f58ea5578e1-kube-api-access-m497g\" (UniqueName: \"kubernetes.io/projected/d0ca0910-85fc-4163-a801-3f58ea5578e1-kube-api-access-m497g\") pod \"d0ca0910-85fc-4163-a801-3f58ea5578e1\" (UID: \"d0ca0910-85fc-4163-a801-3f58ea5578e1\") " Nov 03 20:34:52.365592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.365569 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-dshm\") pod \"a5a5de13-728b-4a75-8990-807d1d3811e4\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " Nov 03 20:34:52.365829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.365801 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a5de13-728b-4a75-8990-807d1d3811e4-kube-api-access-h5krm\" (UniqueName: \"kubernetes.io/projected/a5a5de13-728b-4a75-8990-807d1d3811e4-kube-api-access-h5krm\") pod \"a5a5de13-728b-4a75-8990-807d1d3811e4\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " Nov 03 20:34:52.366036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.366018 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-tmp-dir\") pod \"a5a5de13-728b-4a75-8990-807d1d3811e4\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " Nov 03 20:34:52.366160 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.366145 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-model-cache\") pod \"a5a5de13-728b-4a75-8990-807d1d3811e4\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " Nov 03 20:34:52.366306 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.366290 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-xks-ca\") pod \"d0ca0910-85fc-4163-a801-3f58ea5578e1\" (UID: \"d0ca0910-85fc-4163-a801-3f58ea5578e1\") " Nov 03 20:34:52.366465 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.366448 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-home\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-home\") pod \"a5a5de13-728b-4a75-8990-807d1d3811e4\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " Nov 03 20:34:52.366619 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.366602 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5a5de13-728b-4a75-8990-807d1d3811e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a5de13-728b-4a75-8990-807d1d3811e4-tls-certs\") pod \"a5a5de13-728b-4a75-8990-807d1d3811e4\" (UID: \"a5a5de13-728b-4a75-8990-807d1d3811e4\") " Nov 03 20:34:52.367330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.367309 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-openshift-ca\") pod \"d0ca0910-85fc-4163-a801-3f58ea5578e1\" (UID: \"d0ca0910-85fc-4163-a801-3f58ea5578e1\") " Nov 03 20:34:52.368031 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.368009 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.368164 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.368149 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/008491db-6f31-4ea1-9cbf-f9f779af78ea-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.368260 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.368248 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.368349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.368338 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.368539 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.368522 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/008491db-6f31-4ea1-9cbf-f9f779af78ea-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.368657 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.368640 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v28h5\" (UniqueName: \"kubernetes.io/projected/008491db-6f31-4ea1-9cbf-f9f779af78ea-kube-api-access-v28h5\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.370327 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.370027 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-tmp-dir" pod "a5a5de13-728b-4a75-8990-807d1d3811e4" (UID: "a5a5de13-728b-4a75-8990-807d1d3811e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.371200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.371179 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a5de13-728b-4a75-8990-807d1d3811e4-tls-certs" pod "a5a5de13-728b-4a75-8990-807d1d3811e4" (UID: "a5a5de13-728b-4a75-8990-807d1d3811e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:52.371612 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.371591 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-home" pod "a5a5de13-728b-4a75-8990-807d1d3811e4" (UID: "a5a5de13-728b-4a75-8990-807d1d3811e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.371926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.371904 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-model-cache" pod "a5a5de13-728b-4a75-8990-807d1d3811e4" (UID: "a5a5de13-728b-4a75-8990-807d1d3811e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.372441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.372418 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-openshift-ca" pod "d0ca0910-85fc-4163-a801-3f58ea5578e1" (UID: "d0ca0910-85fc-4163-a801-3f58ea5578e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:52.386574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.384817 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-xks-ca" pod "d0ca0910-85fc-4163-a801-3f58ea5578e1" (UID: "d0ca0910-85fc-4163-a801-3f58ea5578e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:34:52.412760 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.412697 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a5de13-728b-4a75-8990-807d1d3811e4-kube-api-access-h5krm" pod "a5a5de13-728b-4a75-8990-807d1d3811e4" (UID: "a5a5de13-728b-4a75-8990-807d1d3811e4"). InnerVolumeSpecName "kube-api-access-h5krm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:52.433671 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.429568 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-dshm" pod "a5a5de13-728b-4a75-8990-807d1d3811e4" (UID: "a5a5de13-728b-4a75-8990-807d1d3811e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:52.441127 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.441093 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ca0910-85fc-4163-a801-3f58ea5578e1-kube-api-access-m497g" pod "d0ca0910-85fc-4163-a801-3f58ea5578e1" (UID: "d0ca0910-85fc-4163-a801-3f58ea5578e1"). InnerVolumeSpecName "kube-api-access-m497g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:52.470988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.470812 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.470988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.470851 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m497g\" (UniqueName: \"kubernetes.io/projected/d0ca0910-85fc-4163-a801-3f58ea5578e1-kube-api-access-m497g\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.470988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.470866 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.470988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.470883 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5krm\" (UniqueName: \"kubernetes.io/projected/a5a5de13-728b-4a75-8990-807d1d3811e4-kube-api-access-h5krm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.470988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.470897 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.470988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.470911 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.470988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.470924 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0ca0910-85fc-4163-a801-3f58ea5578e1-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.470988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.470938 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5a5de13-728b-4a75-8990-807d1d3811e4-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.470988 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.470951 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5a5de13-728b-4a75-8990-807d1d3811e4-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:52.735595 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.735462 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst"] Nov 03 20:34:52.740453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:52.740417 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a744eb-6tbst"] Nov 03 20:34:53.087651 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.087622 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" Nov 03 20:34:53.088145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.087653 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps" event={"ID":"d0ca0910-85fc-4163-a801-3f58ea5578e1","Type":"ContainerDied","Data":"873ef430dda19f0db17cdefd97202350445ed0c40406e8c484fe903fc1452966"} Nov 03 20:34:53.088145 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.087712 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="873ef430dda19f0db17cdefd97202350445ed0c40406e8c484fe903fc1452966" Nov 03 20:34:53.092426 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.089873 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" Nov 03 20:34:53.092426 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.089880 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp" event={"ID":"008491db-6f31-4ea1-9cbf-f9f779af78ea","Type":"ContainerDied","Data":"b9da716f0beeec0533a452557595f5420b9c96f6946725689be676f4ce23a871"} Nov 03 20:34:53.092426 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.090036 4457 scope.go:122] "RemoveContainer" containerID="2bfec98d165f28e2a730aa0a33f5a026f669cb0b9cea36b537d87bf37a6208d3" Nov 03 20:34:53.094059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.094029 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" event={"ID":"a5a5de13-728b-4a75-8990-807d1d3811e4","Type":"ContainerDied","Data":"ea42f29c37048574987e18c348b015b9f4fea853009dd05b59245a69df9507ab"} Nov 03 20:34:53.094249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.094112 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq" Nov 03 20:34:53.111539 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.111487 4457 scope.go:122] "RemoveContainer" containerID="a4863bcc5fe6ab9849a22ebb0926c64f321e44e4d6b2a3949c215f2ef1b85d25" Nov 03 20:34:53.111722 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.111690 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" podUID="1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f" containerName="payload-pre-processing" containerID="cri-o://5b8310b3086cae90cbcb030e36a6cbee22bb8291b5b8a824d397cfe07f2f1ce4" gracePeriod=25 Nov 03 20:34:53.127554 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.127516 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp"] Nov 03 20:34:53.133475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.133441 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/distinct-w6-d412cf-kserve-6dff556f-mgbsp"] Nov 03 20:34:53.173728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.164825 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq"] Nov 03 20:34:53.173728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.172220 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857"] Nov 03 20:34:53.173728 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.172484 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerName="istio-proxy" containerID="cri-o://9950a33eb973d7f9a27b8fa158bc4e85656e66a1298f5bf280da89a65466943d" gracePeriod=30 Nov 03 20:34:53.178532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.178506 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d412cf/simulator-w6-d412cf-kserve-b979cf7bc-zfwnq"] Nov 03 20:34:53.758844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.758800 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" path="/var/lib/kubelet/pods/008491db-6f31-4ea1-9cbf-f9f779af78ea/volumes" Nov 03 20:34:53.759382 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.759359 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" path="/var/lib/kubelet/pods/a5a5de13-728b-4a75-8990-807d1d3811e4/volumes" Nov 03 20:34:53.759922 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.759900 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8cce7c2-a80e-4336-aca9-f61c47586e38" path="/var/lib/kubelet/pods/f8cce7c2-a80e-4336-aca9-f61c47586e38/volumes" Nov 03 20:34:53.974045 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:53.974001 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" podUID="1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:34:54.590377 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:54.590322 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb"] Nov 03 20:34:54.591211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:54.591151 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" podUID="92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" containerName="istio-proxy" containerID="cri-o://da60cc90637ea437b3c8d97c02b9fdbb1e2727789243830d722351d1890d204f" gracePeriod=30 Nov 03 20:34:54.600926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:54.596211 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs"] Nov 03 20:34:54.600926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:54.596429 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" podUID="7fa9c354-722f-46a0-be30-c4683f7e2254" containerName="istio-proxy" containerID="cri-o://83594e7b8b89929813990979f540dfc2732b338a59fabac9756e4bd7f62b979b" gracePeriod=30 Nov 03 20:34:57.373688 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:57.373648 4457 patch_prober.go:29] interesting pod/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:34:57.374158 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:57.373712 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:34:58.467344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.467296 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:34:58.582169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.582130 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-credential-socket\") pod \"a6b8b440-7a7c-4713-bce7-529cf543c29d\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " Nov 03 20:34:58.582340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.582261 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-kube-api-access-24jqv\" (UniqueName: \"kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-kube-api-access-24jqv\") pod \"a6b8b440-7a7c-4713-bce7-529cf543c29d\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " Nov 03 20:34:58.582340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.582303 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-certs\") pod \"a6b8b440-7a7c-4713-bce7-529cf543c29d\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " Nov 03 20:34:58.582532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.582348 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6b8b440-7a7c-4713-bce7-529cf543c29d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b8b440-7a7c-4713-bce7-529cf543c29d-istiod-ca-cert\") pod \"a6b8b440-7a7c-4713-bce7-529cf543c29d\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " Nov 03 20:34:58.582532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.582375 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-data\") pod \"a6b8b440-7a7c-4713-bce7-529cf543c29d\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " Nov 03 20:34:58.582532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.582465 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-socket\") pod \"a6b8b440-7a7c-4713-bce7-529cf543c29d\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " Nov 03 20:34:58.582532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.582484 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-credential-socket" pod "a6b8b440-7a7c-4713-bce7-529cf543c29d" (UID: "a6b8b440-7a7c-4713-bce7-529cf543c29d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:58.582532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.582516 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-envoy\") pod \"a6b8b440-7a7c-4713-bce7-529cf543c29d\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " Nov 03 20:34:58.582817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.582560 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-podinfo\") pod \"a6b8b440-7a7c-4713-bce7-529cf543c29d\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " Nov 03 20:34:58.582817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.582617 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-token\" (UniqueName: \"kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-token\") pod \"a6b8b440-7a7c-4713-bce7-529cf543c29d\" (UID: \"a6b8b440-7a7c-4713-bce7-529cf543c29d\") " Nov 03 20:34:58.582817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.582718 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-data" pod "a6b8b440-7a7c-4713-bce7-529cf543c29d" (UID: "a6b8b440-7a7c-4713-bce7-529cf543c29d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:58.582990 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.582764 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b8b440-7a7c-4713-bce7-529cf543c29d-istiod-ca-cert" pod "a6b8b440-7a7c-4713-bce7-529cf543c29d" (UID: "a6b8b440-7a7c-4713-bce7-529cf543c29d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:34:58.583103 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.583068 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-socket" pod "a6b8b440-7a7c-4713-bce7-529cf543c29d" (UID: "a6b8b440-7a7c-4713-bce7-529cf543c29d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:58.583195 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.583131 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-certs" pod "a6b8b440-7a7c-4713-bce7-529cf543c29d" (UID: "a6b8b440-7a7c-4713-bce7-529cf543c29d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:58.583312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.583295 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:58.583369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.583314 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:58.583369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.583325 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b8b440-7a7c-4713-bce7-529cf543c29d-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:58.583369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.583340 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:58.583369 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.583354 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:58.584933 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.584907 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-podinfo" pod "a6b8b440-7a7c-4713-bce7-529cf543c29d" (UID: "a6b8b440-7a7c-4713-bce7-529cf543c29d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:34:58.585141 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.585114 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-kube-api-access-24jqv" pod "a6b8b440-7a7c-4713-bce7-529cf543c29d" (UID: "a6b8b440-7a7c-4713-bce7-529cf543c29d"). InnerVolumeSpecName "kube-api-access-24jqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:58.585268 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.585246 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-envoy" pod "a6b8b440-7a7c-4713-bce7-529cf543c29d" (UID: "a6b8b440-7a7c-4713-bce7-529cf543c29d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:34:58.585333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.585315 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-token" pod "a6b8b440-7a7c-4713-bce7-529cf543c29d" (UID: "a6b8b440-7a7c-4713-bce7-529cf543c29d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:34:58.674461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.672826 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q"] Nov 03 20:34:58.684916 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.684880 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24jqv\" (UniqueName: \"kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-kube-api-access-24jqv\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:58.684916 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.684920 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:58.685128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.684934 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:58.685128 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:58.684956 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b8b440-7a7c-4713-bce7-529cf543c29d-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:34:59.164118 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.164070 4457 generic.go:356] "Generic (PLEG): container finished" podID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerID="9950a33eb973d7f9a27b8fa158bc4e85656e66a1298f5bf280da89a65466943d" exitCode=0 Nov 03 20:34:59.164335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.164136 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" event={"ID":"a6b8b440-7a7c-4713-bce7-529cf543c29d","Type":"ContainerDied","Data":"9950a33eb973d7f9a27b8fa158bc4e85656e66a1298f5bf280da89a65466943d"} Nov 03 20:34:59.164335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.164150 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" Nov 03 20:34:59.164335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.164221 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857" event={"ID":"a6b8b440-7a7c-4713-bce7-529cf543c29d","Type":"ContainerDied","Data":"56d34024a0cc1cceb6b24a551f8ee027b010fe3ef0232765592c3c58c1d94a65"} Nov 03 20:34:59.164335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.164259 4457 scope.go:122] "RemoveContainer" containerID="9950a33eb973d7f9a27b8fa158bc4e85656e66a1298f5bf280da89a65466943d" Nov 03 20:34:59.183249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.183226 4457 scope.go:122] "RemoveContainer" containerID="39f18f1f4d7bd1e8eed7adb3f8bb1f04487d8c14fdeb36336acd37e30985b78e" Nov 03 20:34:59.199605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.199581 4457 scope.go:122] "RemoveContainer" containerID="9950a33eb973d7f9a27b8fa158bc4e85656e66a1298f5bf280da89a65466943d" Nov 03 20:34:59.199982 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:59.199956 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9950a33eb973d7f9a27b8fa158bc4e85656e66a1298f5bf280da89a65466943d\": container with ID starting with 9950a33eb973d7f9a27b8fa158bc4e85656e66a1298f5bf280da89a65466943d not found: ID does not exist" containerID="9950a33eb973d7f9a27b8fa158bc4e85656e66a1298f5bf280da89a65466943d" Nov 03 20:34:59.200094 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.199993 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9950a33eb973d7f9a27b8fa158bc4e85656e66a1298f5bf280da89a65466943d"} err="failed to get container status \"9950a33eb973d7f9a27b8fa158bc4e85656e66a1298f5bf280da89a65466943d\": rpc error: code = NotFound desc = could not find container \"9950a33eb973d7f9a27b8fa158bc4e85656e66a1298f5bf280da89a65466943d\": container with ID starting with 9950a33eb973d7f9a27b8fa158bc4e85656e66a1298f5bf280da89a65466943d not found: ID does not exist" Nov 03 20:34:59.200094 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.200019 4457 scope.go:122] "RemoveContainer" containerID="39f18f1f4d7bd1e8eed7adb3f8bb1f04487d8c14fdeb36336acd37e30985b78e" Nov 03 20:34:59.200380 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:34:59.200361 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f18f1f4d7bd1e8eed7adb3f8bb1f04487d8c14fdeb36336acd37e30985b78e\": container with ID starting with 39f18f1f4d7bd1e8eed7adb3f8bb1f04487d8c14fdeb36336acd37e30985b78e not found: ID does not exist" containerID="39f18f1f4d7bd1e8eed7adb3f8bb1f04487d8c14fdeb36336acd37e30985b78e" Nov 03 20:34:59.200380 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.200423 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f18f1f4d7bd1e8eed7adb3f8bb1f04487d8c14fdeb36336acd37e30985b78e"} err="failed to get container status \"39f18f1f4d7bd1e8eed7adb3f8bb1f04487d8c14fdeb36336acd37e30985b78e\": rpc error: code = NotFound desc = could not find container \"39f18f1f4d7bd1e8eed7adb3f8bb1f04487d8c14fdeb36336acd37e30985b78e\": container with ID starting with 39f18f1f4d7bd1e8eed7adb3f8bb1f04487d8c14fdeb36336acd37e30985b78e not found: ID does not exist" Nov 03 20:34:59.203769 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.203746 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857"] Nov 03 20:34:59.208670 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.208645 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-1c6a98-openshift-default-55cf494d97-bj857"] Nov 03 20:34:59.759204 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.759165 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" path="/var/lib/kubelet/pods/a6b8b440-7a7c-4713-bce7-529cf543c29d/volumes" Nov 03 20:34:59.909100 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.909070 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:34:59.932259 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:34:59.932222 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008404 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-certs\") pod \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008483 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-credential-socket\") pod \"7fa9c354-722f-46a0-be30-c4683f7e2254\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008512 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa9c354-722f-46a0-be30-c4683f7e2254-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa9c354-722f-46a0-be30-c4683f7e2254-istiod-ca-cert\") pod \"7fa9c354-722f-46a0-be30-c4683f7e2254\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008535 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-envoy\") pod \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008553 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-kube-api-access-pzlzv\" (UniqueName: \"kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-kube-api-access-pzlzv\") pod \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008598 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-envoy\") pod \"7fa9c354-722f-46a0-be30-c4683f7e2254\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008616 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-token\" (UniqueName: \"kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-token\") pod \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008643 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-credential-socket\") pod \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008661 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-podinfo\") pod \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008685 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-data\") pod \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008704 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-podinfo\") pod \"7fa9c354-722f-46a0-be30-c4683f7e2254\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008726 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-certs\") pod \"7fa9c354-722f-46a0-be30-c4683f7e2254\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008759 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-socket\") pod \"7fa9c354-722f-46a0-be30-c4683f7e2254\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008782 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-data\") pod \"7fa9c354-722f-46a0-be30-c4683f7e2254\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008828 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-kube-api-access-jsp82\" (UniqueName: \"kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-kube-api-access-jsp82\") pod \"7fa9c354-722f-46a0-be30-c4683f7e2254\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008909 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-socket\") pod \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008957 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istiod-ca-cert\") pod \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\" (UID: \"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab\") " Nov 03 20:35:00.009210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.008981 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-token\" (UniqueName: \"kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-token\") pod \"7fa9c354-722f-46a0-be30-c4683f7e2254\" (UID: \"7fa9c354-722f-46a0-be30-c4683f7e2254\") " Nov 03 20:35:00.010359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.009944 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-credential-socket" pod "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" (UID: "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.010359 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.010329 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-data" pod "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" (UID: "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.013756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.011086 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-socket" pod "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" (UID: "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.013756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.011835 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-token" pod "7fa9c354-722f-46a0-be30-c4683f7e2254" (UID: "7fa9c354-722f-46a0-be30-c4683f7e2254"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:00.013756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.012579 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-envoy" pod "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" (UID: "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.013756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.013071 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-token" pod "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" (UID: "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:00.013756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.013437 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-kube-api-access-pzlzv" pod "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" (UID: "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab"). InnerVolumeSpecName "kube-api-access-pzlzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:00.013756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.013557 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-podinfo" pod "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" (UID: "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:00.013756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.013717 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istiod-ca-cert" pod "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" (UID: "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:00.014275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.013868 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-certs" pod "7fa9c354-722f-46a0-be30-c4683f7e2254" (UID: "7fa9c354-722f-46a0-be30-c4683f7e2254"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.014275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.013988 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-certs" pod "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" (UID: "92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.014275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.013778 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa9c354-722f-46a0-be30-c4683f7e2254-istiod-ca-cert" pod "7fa9c354-722f-46a0-be30-c4683f7e2254" (UID: "7fa9c354-722f-46a0-be30-c4683f7e2254"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:00.014275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.014063 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-data" pod "7fa9c354-722f-46a0-be30-c4683f7e2254" (UID: "7fa9c354-722f-46a0-be30-c4683f7e2254"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.014275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.013953 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-socket" pod "7fa9c354-722f-46a0-be30-c4683f7e2254" (UID: "7fa9c354-722f-46a0-be30-c4683f7e2254"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.014275 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.014141 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-credential-socket" pod "7fa9c354-722f-46a0-be30-c4683f7e2254" (UID: "7fa9c354-722f-46a0-be30-c4683f7e2254"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.015638 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.015611 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-podinfo" pod "7fa9c354-722f-46a0-be30-c4683f7e2254" (UID: "7fa9c354-722f-46a0-be30-c4683f7e2254"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:00.015805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.015774 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-envoy" pod "7fa9c354-722f-46a0-be30-c4683f7e2254" (UID: "7fa9c354-722f-46a0-be30-c4683f7e2254"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:00.015805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.015796 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-kube-api-access-jsp82" pod "7fa9c354-722f-46a0-be30-c4683f7e2254" (UID: "7fa9c354-722f-46a0-be30-c4683f7e2254"). InnerVolumeSpecName "kube-api-access-jsp82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:00.110296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110207 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsp82\" (UniqueName: \"kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-kube-api-access-jsp82\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110242 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110253 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110262 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110271 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110280 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110289 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fa9c354-722f-46a0-be30-c4683f7e2254-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110297 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110306 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzlzv\" (UniqueName: \"kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-kube-api-access-pzlzv\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110314 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110322 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110331 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110339 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110347 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110358 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110366 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110376 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.110726 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.110399 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fa9c354-722f-46a0-be30-c4683f7e2254-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:00.178432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.177271 4457 generic.go:356] "Generic (PLEG): container finished" podID="92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" containerID="da60cc90637ea437b3c8d97c02b9fdbb1e2727789243830d722351d1890d204f" exitCode=0 Nov 03 20:35:00.178432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.177321 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" Nov 03 20:35:00.178432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.177317 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" event={"ID":"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab","Type":"ContainerDied","Data":"da60cc90637ea437b3c8d97c02b9fdbb1e2727789243830d722351d1890d204f"} Nov 03 20:35:00.178432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.177412 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb" event={"ID":"92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab","Type":"ContainerDied","Data":"b08cd45e6c018a1a6b4967bb9f31198f795f4fe72cb96c4ea624fb29611ea6ee"} Nov 03 20:35:00.178432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.177446 4457 scope.go:122] "RemoveContainer" containerID="da60cc90637ea437b3c8d97c02b9fdbb1e2727789243830d722351d1890d204f" Nov 03 20:35:00.180596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.180360 4457 generic.go:356] "Generic (PLEG): container finished" podID="7fa9c354-722f-46a0-be30-c4683f7e2254" containerID="83594e7b8b89929813990979f540dfc2732b338a59fabac9756e4bd7f62b979b" exitCode=0 Nov 03 20:35:00.180596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.180427 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" Nov 03 20:35:00.180596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.180429 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" event={"ID":"7fa9c354-722f-46a0-be30-c4683f7e2254","Type":"ContainerDied","Data":"83594e7b8b89929813990979f540dfc2732b338a59fabac9756e4bd7f62b979b"} Nov 03 20:35:00.180596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.180502 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs" event={"ID":"7fa9c354-722f-46a0-be30-c4683f7e2254","Type":"ContainerDied","Data":"1828f03b8b0164cee1e50db68efb2599b568b33d52a34d2701c3d3a7c97b27b8"} Nov 03 20:35:00.199537 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.199514 4457 scope.go:122] "RemoveContainer" containerID="da60cc90637ea437b3c8d97c02b9fdbb1e2727789243830d722351d1890d204f" Nov 03 20:35:00.199829 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:00.199809 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da60cc90637ea437b3c8d97c02b9fdbb1e2727789243830d722351d1890d204f\": container with ID starting with da60cc90637ea437b3c8d97c02b9fdbb1e2727789243830d722351d1890d204f not found: ID does not exist" containerID="da60cc90637ea437b3c8d97c02b9fdbb1e2727789243830d722351d1890d204f" Nov 03 20:35:00.199938 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.199841 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da60cc90637ea437b3c8d97c02b9fdbb1e2727789243830d722351d1890d204f"} err="failed to get container status \"da60cc90637ea437b3c8d97c02b9fdbb1e2727789243830d722351d1890d204f\": rpc error: code = NotFound desc = could not find container \"da60cc90637ea437b3c8d97c02b9fdbb1e2727789243830d722351d1890d204f\": container with ID starting with da60cc90637ea437b3c8d97c02b9fdbb1e2727789243830d722351d1890d204f not found: ID does not exist" Nov 03 20:35:00.199938 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.199865 4457 scope.go:122] "RemoveContainer" containerID="83594e7b8b89929813990979f540dfc2732b338a59fabac9756e4bd7f62b979b" Nov 03 20:35:00.211579 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.211555 4457 scope.go:122] "RemoveContainer" containerID="83594e7b8b89929813990979f540dfc2732b338a59fabac9756e4bd7f62b979b" Nov 03 20:35:00.211877 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:00.211856 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83594e7b8b89929813990979f540dfc2732b338a59fabac9756e4bd7f62b979b\": container with ID starting with 83594e7b8b89929813990979f540dfc2732b338a59fabac9756e4bd7f62b979b not found: ID does not exist" containerID="83594e7b8b89929813990979f540dfc2732b338a59fabac9756e4bd7f62b979b" Nov 03 20:35:00.211934 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.211884 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83594e7b8b89929813990979f540dfc2732b338a59fabac9756e4bd7f62b979b"} err="failed to get container status \"83594e7b8b89929813990979f540dfc2732b338a59fabac9756e4bd7f62b979b\": rpc error: code = NotFound desc = could not find container \"83594e7b8b89929813990979f540dfc2732b338a59fabac9756e4bd7f62b979b\": container with ID starting with 83594e7b8b89929813990979f540dfc2732b338a59fabac9756e4bd7f62b979b not found: ID does not exist" Nov 03 20:35:00.215117 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.215086 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb"] Nov 03 20:35:00.219045 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.219023 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-khfrb"] Nov 03 20:35:00.241900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.237578 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs"] Nov 03 20:35:00.245261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.245217 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-62221d-openshift-default-7786c68bd5-qzvjs"] Nov 03 20:35:00.509075 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.508984 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww"] Nov 03 20:35:00.509273 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:00.509242 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" containerID="cri-o://f22f5bf477b46a10362a194352a976a09502728dad81735d18865018fd129d48" gracePeriod=30 Nov 03 20:35:01.758699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:01.758656 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa9c354-722f-46a0-be30-c4683f7e2254" path="/var/lib/kubelet/pods/7fa9c354-722f-46a0-be30-c4683f7e2254/volumes" Nov 03 20:35:01.759330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:01.759308 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" path="/var/lib/kubelet/pods/92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab/volumes" Nov 03 20:35:03.968656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:03.968615 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" podUID="1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:35:04.026214 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:04.026175 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l"] Nov 03 20:35:04.039910 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:04.039877 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g"] Nov 03 20:35:05.893365 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:05.893332 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:35:05.979978 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:05.979934 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g"] Nov 03 20:35:05.986805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:05.986774 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw"] Nov 03 20:35:06.004490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.004455 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-kube-api-access-8ff7d\" (UniqueName: \"kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-kube-api-access-8ff7d\") pod \"7d097968-5008-4658-a534-fe81ba625490\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " Nov 03 20:35:06.004691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.004517 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-socket\") pod \"7d097968-5008-4658-a534-fe81ba625490\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " Nov 03 20:35:06.004691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.004572 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d097968-5008-4658-a534-fe81ba625490-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d097968-5008-4658-a534-fe81ba625490-istiod-ca-cert\") pod \"7d097968-5008-4658-a534-fe81ba625490\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " Nov 03 20:35:06.004691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.004601 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-credential-socket\") pod \"7d097968-5008-4658-a534-fe81ba625490\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " Nov 03 20:35:06.004691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.004629 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-data\") pod \"7d097968-5008-4658-a534-fe81ba625490\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " Nov 03 20:35:06.004921 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.004753 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-certs\") pod \"7d097968-5008-4658-a534-fe81ba625490\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " Nov 03 20:35:06.004921 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.004787 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7d097968-5008-4658-a534-fe81ba625490-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d097968-5008-4658-a534-fe81ba625490-istio-podinfo\") pod \"7d097968-5008-4658-a534-fe81ba625490\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " Nov 03 20:35:06.004921 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.004825 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-envoy\") pod \"7d097968-5008-4658-a534-fe81ba625490\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " Nov 03 20:35:06.004921 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.004905 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-istio-token\" (UniqueName: \"kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-istio-token\") pod \"7d097968-5008-4658-a534-fe81ba625490\" (UID: \"7d097968-5008-4658-a534-fe81ba625490\") " Nov 03 20:35:06.005950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.005925 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-credential-socket" pod "7d097968-5008-4658-a534-fe81ba625490" (UID: "7d097968-5008-4658-a534-fe81ba625490"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:06.006284 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.006237 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg"] Nov 03 20:35:06.006577 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.006432 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-socket" pod "7d097968-5008-4658-a534-fe81ba625490" (UID: "7d097968-5008-4658-a534-fe81ba625490"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:06.010939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.006945 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-data" pod "7d097968-5008-4658-a534-fe81ba625490" (UID: "7d097968-5008-4658-a534-fe81ba625490"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:06.010939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.009186 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-certs" pod "7d097968-5008-4658-a534-fe81ba625490" (UID: "7d097968-5008-4658-a534-fe81ba625490"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:06.010939 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.009810 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d097968-5008-4658-a534-fe81ba625490-istiod-ca-cert" pod "7d097968-5008-4658-a534-fe81ba625490" (UID: "7d097968-5008-4658-a534-fe81ba625490"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:06.011691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.011578 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-istio-token" pod "7d097968-5008-4658-a534-fe81ba625490" (UID: "7d097968-5008-4658-a534-fe81ba625490"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:06.011691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.011658 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-kube-api-access-8ff7d" pod "7d097968-5008-4658-a534-fe81ba625490" (UID: "7d097968-5008-4658-a534-fe81ba625490"). InnerVolumeSpecName "kube-api-access-8ff7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:06.011691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.011662 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-envoy" pod "7d097968-5008-4658-a534-fe81ba625490" (UID: "7d097968-5008-4658-a534-fe81ba625490"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:06.013759 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.013737 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d097968-5008-4658-a534-fe81ba625490-istio-podinfo" pod "7d097968-5008-4658-a534-fe81ba625490" (UID: "7d097968-5008-4658-a534-fe81ba625490"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:06.106686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.106641 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d097968-5008-4658-a534-fe81ba625490-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:06.106686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.106682 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:06.106686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.106691 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:06.106686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.106700 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:06.107116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.106708 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d097968-5008-4658-a534-fe81ba625490-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:06.107116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.106719 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:06.107116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.106728 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:06.107116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.106737 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ff7d\" (UniqueName: \"kubernetes.io/projected/7d097968-5008-4658-a534-fe81ba625490-kube-api-access-8ff7d\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:06.107116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.106748 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d097968-5008-4658-a534-fe81ba625490-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:06.158305 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:06.158210 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:35:06.252685 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.252643 4457 generic.go:356] "Generic (PLEG): container finished" podID="7d097968-5008-4658-a534-fe81ba625490" containerID="f22f5bf477b46a10362a194352a976a09502728dad81735d18865018fd129d48" exitCode=0 Nov 03 20:35:06.252896 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.252698 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" event={"ID":"7d097968-5008-4658-a534-fe81ba625490","Type":"ContainerDied","Data":"f22f5bf477b46a10362a194352a976a09502728dad81735d18865018fd129d48"} Nov 03 20:35:06.252896 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.252713 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" Nov 03 20:35:06.252896 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.252753 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww" event={"ID":"7d097968-5008-4658-a534-fe81ba625490","Type":"ContainerDied","Data":"76ab52b7ff26c06813b6ba5c20234b0573dd5e63764223ef331ef30e582f14e6"} Nov 03 20:35:06.252896 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.252772 4457 scope.go:122] "RemoveContainer" containerID="f22f5bf477b46a10362a194352a976a09502728dad81735d18865018fd129d48" Nov 03 20:35:06.279648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.279602 4457 scope.go:122] "RemoveContainer" containerID="4496f6ade23dd61b318645f2d4d985651b69df45c44dbf2265a9dec503169dfc" Nov 03 20:35:06.301858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.301808 4457 scope.go:122] "RemoveContainer" containerID="f22f5bf477b46a10362a194352a976a09502728dad81735d18865018fd129d48" Nov 03 20:35:06.302759 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:06.302714 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22f5bf477b46a10362a194352a976a09502728dad81735d18865018fd129d48\": container with ID starting with f22f5bf477b46a10362a194352a976a09502728dad81735d18865018fd129d48 not found: ID does not exist" containerID="f22f5bf477b46a10362a194352a976a09502728dad81735d18865018fd129d48" Nov 03 20:35:06.302955 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.302926 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22f5bf477b46a10362a194352a976a09502728dad81735d18865018fd129d48"} err="failed to get container status \"f22f5bf477b46a10362a194352a976a09502728dad81735d18865018fd129d48\": rpc error: code = NotFound desc = could not find container \"f22f5bf477b46a10362a194352a976a09502728dad81735d18865018fd129d48\": container with ID starting with f22f5bf477b46a10362a194352a976a09502728dad81735d18865018fd129d48 not found: ID does not exist" Nov 03 20:35:06.303324 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.303303 4457 scope.go:122] "RemoveContainer" containerID="4496f6ade23dd61b318645f2d4d985651b69df45c44dbf2265a9dec503169dfc" Nov 03 20:35:06.303724 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:06.303694 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4496f6ade23dd61b318645f2d4d985651b69df45c44dbf2265a9dec503169dfc\": container with ID starting with 4496f6ade23dd61b318645f2d4d985651b69df45c44dbf2265a9dec503169dfc not found: ID does not exist" containerID="4496f6ade23dd61b318645f2d4d985651b69df45c44dbf2265a9dec503169dfc" Nov 03 20:35:06.303826 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.303735 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4496f6ade23dd61b318645f2d4d985651b69df45c44dbf2265a9dec503169dfc"} err="failed to get container status \"4496f6ade23dd61b318645f2d4d985651b69df45c44dbf2265a9dec503169dfc\": rpc error: code = NotFound desc = could not find container \"4496f6ade23dd61b318645f2d4d985651b69df45c44dbf2265a9dec503169dfc\": container with ID starting with 4496f6ade23dd61b318645f2d4d985651b69df45c44dbf2265a9dec503169dfc not found: ID does not exist" Nov 03 20:35:06.314729 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.314691 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww"] Nov 03 20:35:06.318313 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:06.318290 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a744eb-openshift-default-767456cb9f-tc9ww"] Nov 03 20:35:06.318893 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:06.318867 4457 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d097968_5008_4658_a534_fe81ba625490.slice/crio-76ab52b7ff26c06813b6ba5c20234b0573dd5e63764223ef331ef30e582f14e6\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d097968_5008_4658_a534_fe81ba625490.slice\": RecentStats: unable to find data in memory cache]" Nov 03 20:35:07.759265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:07.759225 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d097968-5008-4658-a534-fe81ba625490" path="/var/lib/kubelet/pods/7d097968-5008-4658-a534-fe81ba625490/volumes" Nov 03 20:35:07.943539 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:07.943506 4457 scope.go:122] "RemoveContainer" containerID="7c833f359026c0662af271408141bff5d5201f2466a0b98d2082f84f5fa8f3fc" Nov 03 20:35:07.954371 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:07.954347 4457 scope.go:122] "RemoveContainer" containerID="450815db0b4c6a2538f2f108cdb8481a3f259203961a88a28cd7c5f9cbb1adbd" Nov 03 20:35:09.026725 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:09.026686 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" podUID="4de4b055-a669-4760-ba03-041291c506fd" containerName="payload-processing" containerID="cri-o://b1c4a0757d3567e9ac47634d122b35a2841961058d97410a4e2a6ebd76c2cfcb" gracePeriod=25 Nov 03 20:35:09.040285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:09.040246 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" podUID="250b873d-632c-4169-b357-77f6d8b9e9b6" containerName="payload-pre-processing" containerID="cri-o://d753e476a9da7265c502be8b7c91fcf092965660a7d3193cef0bdca44d3bf257" gracePeriod=25 Nov 03 20:35:13.332808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:13.332772 4457 generic.go:356] "Generic (PLEG): container finished" podID="1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f" containerID="5b8310b3086cae90cbcb030e36a6cbee22bb8291b5b8a824d397cfe07f2f1ce4" exitCode=0 Nov 03 20:35:13.333300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:13.333082 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" event={"ID":"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f","Type":"ContainerDied","Data":"5b8310b3086cae90cbcb030e36a6cbee22bb8291b5b8a824d397cfe07f2f1ce4"} Nov 03 20:35:13.515983 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:13.515957 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" Nov 03 20:35:13.627761 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:13.627724 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-kube-api-access-mcs8g\" (UniqueName: \"kubernetes.io/projected/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-kube-api-access-mcs8g\") pod \"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f\" (UID: \"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f\") " Nov 03 20:35:13.627946 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:13.627877 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-plugins-config-volume\") pod \"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f\" (UID: \"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f\") " Nov 03 20:35:13.628276 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:13.628249 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-plugins-config-volume" pod "1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f" (UID: "1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:13.629950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:13.629920 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-kube-api-access-mcs8g" pod "1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f" (UID: "1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f"). InnerVolumeSpecName "kube-api-access-mcs8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:13.721744 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:13.721561 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" containerID="cri-o://74bcd017eff3d35badc298cc237c4ab009a62f20203a14cf6667da65487b1070" gracePeriod=45 Nov 03 20:35:13.742480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:13.741239 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcs8g\" (UniqueName: \"kubernetes.io/projected/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-kube-api-access-mcs8g\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:13.742480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:13.741281 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.070247 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.070211 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:35:14.147480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.147443 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-model-cache\") pod \"f3f4267e-0c59-4326-972b-b0925e478929\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " Nov 03 20:35:14.147660 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.147559 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f4267e-0c59-4326-972b-b0925e478929-tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f4267e-0c59-4326-972b-b0925e478929-tls-certs\") pod \"f3f4267e-0c59-4326-972b-b0925e478929\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " Nov 03 20:35:14.147660 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.147619 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f4267e-0c59-4326-972b-b0925e478929-kube-api-access-9ss95\" (UniqueName: \"kubernetes.io/projected/f3f4267e-0c59-4326-972b-b0925e478929-kube-api-access-9ss95\") pod \"f3f4267e-0c59-4326-972b-b0925e478929\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " Nov 03 20:35:14.147797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.147661 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-home\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-home\") pod \"f3f4267e-0c59-4326-972b-b0925e478929\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " Nov 03 20:35:14.147797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.147663 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-model-cache" pod "f3f4267e-0c59-4326-972b-b0925e478929" (UID: "f3f4267e-0c59-4326-972b-b0925e478929"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.147797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.147764 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-tmp-dir\") pod \"f3f4267e-0c59-4326-972b-b0925e478929\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " Nov 03 20:35:14.147963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.147798 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-dshm\") pod \"f3f4267e-0c59-4326-972b-b0925e478929\" (UID: \"f3f4267e-0c59-4326-972b-b0925e478929\") " Nov 03 20:35:14.148051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.148010 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-home" pod "f3f4267e-0c59-4326-972b-b0925e478929" (UID: "f3f4267e-0c59-4326-972b-b0925e478929"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.148143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.148052 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-tmp-dir" pod "f3f4267e-0c59-4326-972b-b0925e478929" (UID: "f3f4267e-0c59-4326-972b-b0925e478929"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.148352 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.148312 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.148352 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.148337 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.148352 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.148351 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.150100 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.150075 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f4267e-0c59-4326-972b-b0925e478929-tls-certs" pod "f3f4267e-0c59-4326-972b-b0925e478929" (UID: "f3f4267e-0c59-4326-972b-b0925e478929"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:14.150211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.150129 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f4267e-0c59-4326-972b-b0925e478929-kube-api-access-9ss95" pod "f3f4267e-0c59-4326-972b-b0925e478929" (UID: "f3f4267e-0c59-4326-972b-b0925e478929"). InnerVolumeSpecName "kube-api-access-9ss95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:14.150426 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.150400 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-dshm" pod "f3f4267e-0c59-4326-972b-b0925e478929" (UID: "f3f4267e-0c59-4326-972b-b0925e478929"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:14.249304 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.249263 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ss95\" (UniqueName: \"kubernetes.io/projected/f3f4267e-0c59-4326-972b-b0925e478929-kube-api-access-9ss95\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.249304 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.249304 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f4267e-0c59-4326-972b-b0925e478929-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.249588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.249319 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f4267e-0c59-4326-972b-b0925e478929-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:14.342105 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.342008 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" Nov 03 20:35:14.342105 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.342007 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj" event={"ID":"1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f","Type":"ContainerDied","Data":"0f91ee99c8cab2d2e458b65bcbb8fc0ee3ed91cd69bc183de095f8276ba0f1c9"} Nov 03 20:35:14.342735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.342152 4457 scope.go:122] "RemoveContainer" containerID="5b8310b3086cae90cbcb030e36a6cbee22bb8291b5b8a824d397cfe07f2f1ce4" Nov 03 20:35:14.343962 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.343923 4457 generic.go:356] "Generic (PLEG): container finished" podID="f3f4267e-0c59-4326-972b-b0925e478929" containerID="74bcd017eff3d35badc298cc237c4ab009a62f20203a14cf6667da65487b1070" exitCode=0 Nov 03 20:35:14.344095 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.343984 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" Nov 03 20:35:14.344095 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.344044 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" event={"ID":"f3f4267e-0c59-4326-972b-b0925e478929","Type":"ContainerDied","Data":"74bcd017eff3d35badc298cc237c4ab009a62f20203a14cf6667da65487b1070"} Nov 03 20:35:14.344223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.344092 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q" event={"ID":"f3f4267e-0c59-4326-972b-b0925e478929","Type":"ContainerDied","Data":"ee14ea6bf6f6b6302e14466f6f60949d71fe4e10e143a8a7facd973e748ab962"} Nov 03 20:35:14.354559 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.354537 4457 scope.go:122] "RemoveContainer" containerID="74bcd017eff3d35badc298cc237c4ab009a62f20203a14cf6667da65487b1070" Nov 03 20:35:14.364758 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.364733 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj"] Nov 03 20:35:14.366884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.366865 4457 scope.go:122] "RemoveContainer" containerID="74bcd017eff3d35badc298cc237c4ab009a62f20203a14cf6667da65487b1070" Nov 03 20:35:14.367203 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:14.367185 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bcd017eff3d35badc298cc237c4ab009a62f20203a14cf6667da65487b1070\": container with ID starting with 74bcd017eff3d35badc298cc237c4ab009a62f20203a14cf6667da65487b1070 not found: ID does not exist" containerID="74bcd017eff3d35badc298cc237c4ab009a62f20203a14cf6667da65487b1070" Nov 03 20:35:14.367285 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.367214 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bcd017eff3d35badc298cc237c4ab009a62f20203a14cf6667da65487b1070"} err="failed to get container status \"74bcd017eff3d35badc298cc237c4ab009a62f20203a14cf6667da65487b1070\": rpc error: code = NotFound desc = could not find container \"74bcd017eff3d35badc298cc237c4ab009a62f20203a14cf6667da65487b1070\": container with ID starting with 74bcd017eff3d35badc298cc237c4ab009a62f20203a14cf6667da65487b1070 not found: ID does not exist" Nov 03 20:35:14.369307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.369284 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a744eb-5bb5b5f6c9-pxknj"] Nov 03 20:35:14.384741 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.384713 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q"] Nov 03 20:35:14.390661 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:14.390640 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-3aba31/premium-w4-3aba31-kserve-bc7889789-vsl6q"] Nov 03 20:35:15.757349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:15.757318 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f" path="/var/lib/kubelet/pods/1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f/volumes" Nov 03 20:35:15.757829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:15.757805 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f4267e-0c59-4326-972b-b0925e478929" path="/var/lib/kubelet/pods/f3f4267e-0c59-4326-972b-b0925e478929/volumes" Nov 03 20:35:18.024730 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:18.024686 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" podUID="4de4b055-a669-4760-ba03-041291c506fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:35:18.025147 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:18.024721 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" podUID="250b873d-632c-4169-b357-77f6d8b9e9b6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:35:21.010732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.010681 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" containerID="cri-o://870e7329d87ec9006d0522e43377aaeecb7559197086d2a7b046f0d62274a2af" gracePeriod=45 Nov 03 20:35:21.011830 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.011786 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" containerID="cri-o://a074aca130d8f183d18598261425a6c1295add2b1f78eec7bdefb3b184933c8d" gracePeriod=45 Nov 03 20:35:21.032997 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.032956 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" containerID="cri-o://4a4f8e4fae0f4de61e6de911f894280be7689ae2b72fc94aa74dc2e3d090d592" gracePeriod=45 Nov 03 20:35:21.414109 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.413576 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:35:21.419319 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.419245 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:35:21.422872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.422457 4457 generic.go:356] "Generic (PLEG): container finished" podID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerID="a074aca130d8f183d18598261425a6c1295add2b1f78eec7bdefb3b184933c8d" exitCode=0 Nov 03 20:35:21.422872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.422554 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" event={"ID":"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1","Type":"ContainerDied","Data":"a074aca130d8f183d18598261425a6c1295add2b1f78eec7bdefb3b184933c8d"} Nov 03 20:35:21.422872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.422588 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" event={"ID":"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1","Type":"ContainerDied","Data":"037097b554fa09f844716c78f975f1a7e5b8657e31b38eae5afaa4fb85a326cd"} Nov 03 20:35:21.422872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.422608 4457 scope.go:122] "RemoveContainer" containerID="a074aca130d8f183d18598261425a6c1295add2b1f78eec7bdefb3b184933c8d" Nov 03 20:35:21.422872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.422758 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw" Nov 03 20:35:21.425176 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.425135 4457 generic.go:356] "Generic (PLEG): container finished" podID="04b0efa8-3782-4cab-be92-a05fa383e808" containerID="4a4f8e4fae0f4de61e6de911f894280be7689ae2b72fc94aa74dc2e3d090d592" exitCode=0 Nov 03 20:35:21.425876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.425371 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" Nov 03 20:35:21.426041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.425458 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" event={"ID":"04b0efa8-3782-4cab-be92-a05fa383e808","Type":"ContainerDied","Data":"4a4f8e4fae0f4de61e6de911f894280be7689ae2b72fc94aa74dc2e3d090d592"} Nov 03 20:35:21.426138 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.426062 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg" event={"ID":"04b0efa8-3782-4cab-be92-a05fa383e808","Type":"ContainerDied","Data":"e9b51d7cf82896d014612835462bd8d604e35e55d4faadf206bd647f76610b84"} Nov 03 20:35:21.431094 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.431055 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:35:21.452948 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.452918 4457 generic.go:356] "Generic (PLEG): container finished" podID="22be1486-517e-4055-8e00-c24c944eb960" containerID="870e7329d87ec9006d0522e43377aaeecb7559197086d2a7b046f0d62274a2af" exitCode=0 Nov 03 20:35:21.453127 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.452983 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" event={"ID":"22be1486-517e-4055-8e00-c24c944eb960","Type":"ContainerDied","Data":"870e7329d87ec9006d0522e43377aaeecb7559197086d2a7b046f0d62274a2af"} Nov 03 20:35:21.453127 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.453014 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" event={"ID":"22be1486-517e-4055-8e00-c24c944eb960","Type":"ContainerDied","Data":"5dfb17eaeb2aeb1a47482e8e5ac521d58a0a5a56db1a19ff527e3b535a935f22"} Nov 03 20:35:21.458972 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.458933 4457 scope.go:122] "RemoveContainer" containerID="a074aca130d8f183d18598261425a6c1295add2b1f78eec7bdefb3b184933c8d" Nov 03 20:35:21.459335 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:21.459305 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a074aca130d8f183d18598261425a6c1295add2b1f78eec7bdefb3b184933c8d\": container with ID starting with a074aca130d8f183d18598261425a6c1295add2b1f78eec7bdefb3b184933c8d not found: ID does not exist" containerID="a074aca130d8f183d18598261425a6c1295add2b1f78eec7bdefb3b184933c8d" Nov 03 20:35:21.459474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.459348 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a074aca130d8f183d18598261425a6c1295add2b1f78eec7bdefb3b184933c8d"} err="failed to get container status \"a074aca130d8f183d18598261425a6c1295add2b1f78eec7bdefb3b184933c8d\": rpc error: code = NotFound desc = could not find container \"a074aca130d8f183d18598261425a6c1295add2b1f78eec7bdefb3b184933c8d\": container with ID starting with a074aca130d8f183d18598261425a6c1295add2b1f78eec7bdefb3b184933c8d not found: ID does not exist" Nov 03 20:35:21.459474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.459371 4457 scope.go:122] "RemoveContainer" containerID="4a4f8e4fae0f4de61e6de911f894280be7689ae2b72fc94aa74dc2e3d090d592" Nov 03 20:35:21.473154 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.472769 4457 scope.go:122] "RemoveContainer" containerID="4a4f8e4fae0f4de61e6de911f894280be7689ae2b72fc94aa74dc2e3d090d592" Nov 03 20:35:21.473857 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:21.473681 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4f8e4fae0f4de61e6de911f894280be7689ae2b72fc94aa74dc2e3d090d592\": container with ID starting with 4a4f8e4fae0f4de61e6de911f894280be7689ae2b72fc94aa74dc2e3d090d592 not found: ID does not exist" containerID="4a4f8e4fae0f4de61e6de911f894280be7689ae2b72fc94aa74dc2e3d090d592" Nov 03 20:35:21.473857 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.473719 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4f8e4fae0f4de61e6de911f894280be7689ae2b72fc94aa74dc2e3d090d592"} err="failed to get container status \"4a4f8e4fae0f4de61e6de911f894280be7689ae2b72fc94aa74dc2e3d090d592\": rpc error: code = NotFound desc = could not find container \"4a4f8e4fae0f4de61e6de911f894280be7689ae2b72fc94aa74dc2e3d090d592\": container with ID starting with 4a4f8e4fae0f4de61e6de911f894280be7689ae2b72fc94aa74dc2e3d090d592 not found: ID does not exist" Nov 03 20:35:21.473857 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.473744 4457 scope.go:122] "RemoveContainer" containerID="870e7329d87ec9006d0522e43377aaeecb7559197086d2a7b046f0d62274a2af" Nov 03 20:35:21.507097 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.506987 4457 scope.go:122] "RemoveContainer" containerID="870e7329d87ec9006d0522e43377aaeecb7559197086d2a7b046f0d62274a2af" Nov 03 20:35:21.514929 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:21.514892 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870e7329d87ec9006d0522e43377aaeecb7559197086d2a7b046f0d62274a2af\": container with ID starting with 870e7329d87ec9006d0522e43377aaeecb7559197086d2a7b046f0d62274a2af not found: ID does not exist" containerID="870e7329d87ec9006d0522e43377aaeecb7559197086d2a7b046f0d62274a2af" Nov 03 20:35:21.515170 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.515137 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870e7329d87ec9006d0522e43377aaeecb7559197086d2a7b046f0d62274a2af"} err="failed to get container status \"870e7329d87ec9006d0522e43377aaeecb7559197086d2a7b046f0d62274a2af\": rpc error: code = NotFound desc = could not find container \"870e7329d87ec9006d0522e43377aaeecb7559197086d2a7b046f0d62274a2af\": container with ID starting with 870e7329d87ec9006d0522e43377aaeecb7559197086d2a7b046f0d62274a2af not found: ID does not exist" Nov 03 20:35:21.582310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582280 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-home\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-home\") pod \"22be1486-517e-4055-8e00-c24c944eb960\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " Nov 03 20:35:21.582536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582333 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tmp-dir\") pod \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " Nov 03 20:35:21.582536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582359 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-model-cache\") pod \"04b0efa8-3782-4cab-be92-a05fa383e808\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " Nov 03 20:35:21.582536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582425 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-dshm\") pod \"04b0efa8-3782-4cab-be92-a05fa383e808\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " Nov 03 20:35:21.582536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582450 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-model-cache\") pod \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " Nov 03 20:35:21.582536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582495 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-home\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-home\") pod \"04b0efa8-3782-4cab-be92-a05fa383e808\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " Nov 03 20:35:21.582834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582549 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-dshm\") pod \"22be1486-517e-4055-8e00-c24c944eb960\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " Nov 03 20:35:21.582834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582597 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-tmp-dir\") pod \"04b0efa8-3782-4cab-be92-a05fa383e808\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " Nov 03 20:35:21.582834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582633 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22be1486-517e-4055-8e00-c24c944eb960-kube-api-access-2zqfs\" (UniqueName: \"kubernetes.io/projected/22be1486-517e-4055-8e00-c24c944eb960-kube-api-access-2zqfs\") pod \"22be1486-517e-4055-8e00-c24c944eb960\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " Nov 03 20:35:21.582834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582664 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22be1486-517e-4055-8e00-c24c944eb960-tls-certs\" (UniqueName: \"kubernetes.io/secret/22be1486-517e-4055-8e00-c24c944eb960-tls-certs\") pod \"22be1486-517e-4055-8e00-c24c944eb960\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " Nov 03 20:35:21.582834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582701 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tls-certs\") pod \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " Nov 03 20:35:21.582834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582751 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b0efa8-3782-4cab-be92-a05fa383e808-kube-api-access-pjg92\" (UniqueName: \"kubernetes.io/projected/04b0efa8-3782-4cab-be92-a05fa383e808-kube-api-access-pjg92\") pod \"04b0efa8-3782-4cab-be92-a05fa383e808\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " Nov 03 20:35:21.582834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582784 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-kube-api-access-7v8cr\" (UniqueName: \"kubernetes.io/projected/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-kube-api-access-7v8cr\") pod \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " Nov 03 20:35:21.582834 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582825 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-home\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-home\") pod \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " Nov 03 20:35:21.583272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582851 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-tmp-dir\") pod \"22be1486-517e-4055-8e00-c24c944eb960\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " Nov 03 20:35:21.583272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582905 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-model-cache\") pod \"22be1486-517e-4055-8e00-c24c944eb960\" (UID: \"22be1486-517e-4055-8e00-c24c944eb960\") " Nov 03 20:35:21.583272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.582951 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04b0efa8-3782-4cab-be92-a05fa383e808-tls-certs\" (UniqueName: \"kubernetes.io/secret/04b0efa8-3782-4cab-be92-a05fa383e808-tls-certs\") pod \"04b0efa8-3782-4cab-be92-a05fa383e808\" (UID: \"04b0efa8-3782-4cab-be92-a05fa383e808\") " Nov 03 20:35:21.583272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.583001 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-dshm\") pod \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\" (UID: \"c209ddd2-c195-4bd5-ba52-b6f13b6a35a1\") " Nov 03 20:35:21.583745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.583717 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tmp-dir" pod "c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" (UID: "c209ddd2-c195-4bd5-ba52-b6f13b6a35a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.584055 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.584028 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-model-cache" pod "04b0efa8-3782-4cab-be92-a05fa383e808" (UID: "04b0efa8-3782-4cab-be92-a05fa383e808"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.584146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.584034 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-home" pod "22be1486-517e-4055-8e00-c24c944eb960" (UID: "22be1486-517e-4055-8e00-c24c944eb960"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.584146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.584103 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-tmp-dir" pod "22be1486-517e-4055-8e00-c24c944eb960" (UID: "22be1486-517e-4055-8e00-c24c944eb960"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.584954 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.584304 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-home" pod "04b0efa8-3782-4cab-be92-a05fa383e808" (UID: "04b0efa8-3782-4cab-be92-a05fa383e808"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.584954 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.584317 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-model-cache" pod "22be1486-517e-4055-8e00-c24c944eb960" (UID: "22be1486-517e-4055-8e00-c24c944eb960"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.584954 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.584557 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-home" pod "c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" (UID: "c209ddd2-c195-4bd5-ba52-b6f13b6a35a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.584954 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.584888 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-tmp-dir" pod "04b0efa8-3782-4cab-be92-a05fa383e808" (UID: "04b0efa8-3782-4cab-be92-a05fa383e808"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.586695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.586667 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22be1486-517e-4055-8e00-c24c944eb960-kube-api-access-2zqfs" pod "22be1486-517e-4055-8e00-c24c944eb960" (UID: "22be1486-517e-4055-8e00-c24c944eb960"). InnerVolumeSpecName "kube-api-access-2zqfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:21.587267 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.587241 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-dshm" pod "22be1486-517e-4055-8e00-c24c944eb960" (UID: "22be1486-517e-4055-8e00-c24c944eb960"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.594686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.594659 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-model-cache" pod "c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" (UID: "c209ddd2-c195-4bd5-ba52-b6f13b6a35a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.595534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.595178 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-dshm" pod "c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" (UID: "c209ddd2-c195-4bd5-ba52-b6f13b6a35a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.595534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.595265 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tls-certs" pod "c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" (UID: "c209ddd2-c195-4bd5-ba52-b6f13b6a35a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:21.595534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.595355 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b0efa8-3782-4cab-be92-a05fa383e808-kube-api-access-pjg92" pod "04b0efa8-3782-4cab-be92-a05fa383e808" (UID: "04b0efa8-3782-4cab-be92-a05fa383e808"). InnerVolumeSpecName "kube-api-access-pjg92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:21.595534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.595444 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22be1486-517e-4055-8e00-c24c944eb960-tls-certs" pod "22be1486-517e-4055-8e00-c24c944eb960" (UID: "22be1486-517e-4055-8e00-c24c944eb960"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:21.596666 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.596642 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04b0efa8-3782-4cab-be92-a05fa383e808-tls-certs" pod "04b0efa8-3782-4cab-be92-a05fa383e808" (UID: "04b0efa8-3782-4cab-be92-a05fa383e808"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:21.596745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.596726 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-kube-api-access-7v8cr" pod "c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" (UID: "c209ddd2-c195-4bd5-ba52-b6f13b6a35a1"). InnerVolumeSpecName "kube-api-access-7v8cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:21.596890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.596872 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-dshm" pod "04b0efa8-3782-4cab-be92-a05fa383e808" (UID: "04b0efa8-3782-4cab-be92-a05fa383e808"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:21.684749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684704 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.684749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684745 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684761 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684775 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684794 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684808 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684820 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b0efa8-3782-4cab-be92-a05fa383e808-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684835 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zqfs\" (UniqueName: \"kubernetes.io/projected/22be1486-517e-4055-8e00-c24c944eb960-kube-api-access-2zqfs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684850 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22be1486-517e-4055-8e00-c24c944eb960-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684863 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684875 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjg92\" (UniqueName: \"kubernetes.io/projected/04b0efa8-3782-4cab-be92-a05fa383e808-kube-api-access-pjg92\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684889 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v8cr\" (UniqueName: \"kubernetes.io/projected/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-kube-api-access-7v8cr\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684902 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684917 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684930 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684942 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b0efa8-3782-4cab-be92-a05fa383e808-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684955 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.685037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.684968 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22be1486-517e-4055-8e00-c24c944eb960-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:21.789418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.789375 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg"] Nov 03 20:35:21.793021 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.792996 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/simulator-w5-a6f98b-kserve-c56d8dc68-cr4mg"] Nov 03 20:35:21.802903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.802877 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw"] Nov 03 20:35:21.807511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:21.807472 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/distinct-w5-a6f98b-kserve-9c7556976-jzgtw"] Nov 03 20:35:22.464574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:22.464529 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g" Nov 03 20:35:22.489136 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:22.489076 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g"] Nov 03 20:35:22.492674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:22.492646 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a6f98b/distinct-2-w5-a6f98b-kserve-78786759c9-cg99g"] Nov 03 20:35:23.767230 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:23.767198 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" path="/var/lib/kubelet/pods/04b0efa8-3782-4cab-be92-a05fa383e808/volumes" Nov 03 20:35:23.767802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:23.767786 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22be1486-517e-4055-8e00-c24c944eb960" path="/var/lib/kubelet/pods/22be1486-517e-4055-8e00-c24c944eb960/volumes" Nov 03 20:35:23.768191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:23.768179 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" path="/var/lib/kubelet/pods/c209ddd2-c195-4bd5-ba52-b6f13b6a35a1/volumes" Nov 03 20:35:26.031896 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:26.031078 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j"] Nov 03 20:35:26.040419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:26.040365 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz"] Nov 03 20:35:26.043490 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:26.043462 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ac980d0-x5t5j"] Nov 03 20:35:26.046971 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:26.046944 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aeba7587-8vhhz"] Nov 03 20:35:26.344742 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:26.344705 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh"] Nov 03 20:35:26.345021 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:26.344987 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" podUID="6cf12593-ef2a-4848-be72-ce57d759c2e9" containerName="istio-proxy" containerID="cri-o://b8049778edcb9e1b90bdca5cb8dcf3f227004ba34cc059918b1ed4a6b7e23b32" gracePeriod=30 Nov 03 20:35:27.757611 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:27.757575 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4de79d-18ac-4f57-803e-fb09ddcc8134" path="/var/lib/kubelet/pods/7f4de79d-18ac-4f57-803e-fb09ddcc8134/volumes" Nov 03 20:35:27.758227 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:27.758205 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1023951-0cef-4afe-b7d8-76d8a9775abb" path="/var/lib/kubelet/pods/b1023951-0cef-4afe-b7d8-76d8a9775abb/volumes" Nov 03 20:35:28.026767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:28.026667 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" podUID="250b873d-632c-4169-b357-77f6d8b9e9b6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:35:28.027075 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:28.026681 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" podUID="4de4b055-a669-4760-ba03-041291c506fd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:35:28.071032 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:28.070991 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf"] Nov 03 20:35:28.071525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:28.071492 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" podUID="97f3184c-7322-4684-94e1-9e5e4eecd6e7" containerName="istio-proxy" containerID="cri-o://de1a8c74891e1f40121ffa0dbba05bc979f4d253660c1b97c7bfcd3b45a80b2c" gracePeriod=30 Nov 03 20:35:29.089381 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.089345 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl"] Nov 03 20:35:29.330104 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.330069 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" Nov 03 20:35:29.370452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.370412 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de4b055-a669-4760-ba03-041291c506fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de4b055-a669-4760-ba03-041291c506fd-plugins-config-volume\") pod \"4de4b055-a669-4760-ba03-041291c506fd\" (UID: \"4de4b055-a669-4760-ba03-041291c506fd\") " Nov 03 20:35:29.370696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.370504 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de4b055-a669-4760-ba03-041291c506fd-kube-api-access-vdhch\" (UniqueName: \"kubernetes.io/projected/4de4b055-a669-4760-ba03-041291c506fd-kube-api-access-vdhch\") pod \"4de4b055-a669-4760-ba03-041291c506fd\" (UID: \"4de4b055-a669-4760-ba03-041291c506fd\") " Nov 03 20:35:29.371527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.371496 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de4b055-a669-4760-ba03-041291c506fd-plugins-config-volume" pod "4de4b055-a669-4760-ba03-041291c506fd" (UID: "4de4b055-a669-4760-ba03-041291c506fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:29.375894 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.375864 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de4b055-a669-4760-ba03-041291c506fd-kube-api-access-vdhch" pod "4de4b055-a669-4760-ba03-041291c506fd" (UID: "4de4b055-a669-4760-ba03-041291c506fd"). InnerVolumeSpecName "kube-api-access-vdhch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:29.390196 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.390170 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" Nov 03 20:35:29.471425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.471343 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250b873d-632c-4169-b357-77f6d8b9e9b6-kube-api-access-bv25h\" (UniqueName: \"kubernetes.io/projected/250b873d-632c-4169-b357-77f6d8b9e9b6-kube-api-access-bv25h\") pod \"250b873d-632c-4169-b357-77f6d8b9e9b6\" (UID: \"250b873d-632c-4169-b357-77f6d8b9e9b6\") " Nov 03 20:35:29.471620 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.471581 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/250b873d-632c-4169-b357-77f6d8b9e9b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250b873d-632c-4169-b357-77f6d8b9e9b6-plugins-config-volume\") pod \"250b873d-632c-4169-b357-77f6d8b9e9b6\" (UID: \"250b873d-632c-4169-b357-77f6d8b9e9b6\") " Nov 03 20:35:29.471973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.471950 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4de4b055-a669-4760-ba03-041291c506fd-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:29.472059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.471975 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdhch\" (UniqueName: \"kubernetes.io/projected/4de4b055-a669-4760-ba03-041291c506fd-kube-api-access-vdhch\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:29.472187 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.472049 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250b873d-632c-4169-b357-77f6d8b9e9b6-plugins-config-volume" pod "250b873d-632c-4169-b357-77f6d8b9e9b6" (UID: "250b873d-632c-4169-b357-77f6d8b9e9b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:29.474621 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.474593 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250b873d-632c-4169-b357-77f6d8b9e9b6-kube-api-access-bv25h" pod "250b873d-632c-4169-b357-77f6d8b9e9b6" (UID: "250b873d-632c-4169-b357-77f6d8b9e9b6"). InnerVolumeSpecName "kube-api-access-bv25h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:29.529166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.529129 4457 generic.go:356] "Generic (PLEG): container finished" podID="250b873d-632c-4169-b357-77f6d8b9e9b6" containerID="d753e476a9da7265c502be8b7c91fcf092965660a7d3193cef0bdca44d3bf257" exitCode=0 Nov 03 20:35:29.529344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.529182 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" Nov 03 20:35:29.529344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.529212 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" event={"ID":"250b873d-632c-4169-b357-77f6d8b9e9b6","Type":"ContainerDied","Data":"d753e476a9da7265c502be8b7c91fcf092965660a7d3193cef0bdca44d3bf257"} Nov 03 20:35:29.529344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.529269 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g" event={"ID":"250b873d-632c-4169-b357-77f6d8b9e9b6","Type":"ContainerDied","Data":"627a83be6254c7676c30a4329aaf9075c91de4264af8017f393034259032651e"} Nov 03 20:35:29.529344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.529298 4457 scope.go:122] "RemoveContainer" containerID="d753e476a9da7265c502be8b7c91fcf092965660a7d3193cef0bdca44d3bf257" Nov 03 20:35:29.531628 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.531588 4457 generic.go:356] "Generic (PLEG): container finished" podID="4de4b055-a669-4760-ba03-041291c506fd" containerID="b1c4a0757d3567e9ac47634d122b35a2841961058d97410a4e2a6ebd76c2cfcb" exitCode=0 Nov 03 20:35:29.531765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.531636 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" Nov 03 20:35:29.531765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.531702 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" event={"ID":"4de4b055-a669-4760-ba03-041291c506fd","Type":"ContainerDied","Data":"b1c4a0757d3567e9ac47634d122b35a2841961058d97410a4e2a6ebd76c2cfcb"} Nov 03 20:35:29.531918 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.531771 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l" event={"ID":"4de4b055-a669-4760-ba03-041291c506fd","Type":"ContainerDied","Data":"0976897cd8e00e948dac27c0c467af50a36a0501b99b9e44fa33a4f47d5c90c8"} Nov 03 20:35:29.546486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.546463 4457 scope.go:122] "RemoveContainer" containerID="d753e476a9da7265c502be8b7c91fcf092965660a7d3193cef0bdca44d3bf257" Nov 03 20:35:29.546839 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:29.546816 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d753e476a9da7265c502be8b7c91fcf092965660a7d3193cef0bdca44d3bf257\": container with ID starting with d753e476a9da7265c502be8b7c91fcf092965660a7d3193cef0bdca44d3bf257 not found: ID does not exist" containerID="d753e476a9da7265c502be8b7c91fcf092965660a7d3193cef0bdca44d3bf257" Nov 03 20:35:29.546965 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.546850 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d753e476a9da7265c502be8b7c91fcf092965660a7d3193cef0bdca44d3bf257"} err="failed to get container status \"d753e476a9da7265c502be8b7c91fcf092965660a7d3193cef0bdca44d3bf257\": rpc error: code = NotFound desc = could not find container \"d753e476a9da7265c502be8b7c91fcf092965660a7d3193cef0bdca44d3bf257\": container with ID starting with d753e476a9da7265c502be8b7c91fcf092965660a7d3193cef0bdca44d3bf257 not found: ID does not exist" Nov 03 20:35:29.546965 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.546875 4457 scope.go:122] "RemoveContainer" containerID="b1c4a0757d3567e9ac47634d122b35a2841961058d97410a4e2a6ebd76c2cfcb" Nov 03 20:35:29.560558 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.560535 4457 scope.go:122] "RemoveContainer" containerID="b1c4a0757d3567e9ac47634d122b35a2841961058d97410a4e2a6ebd76c2cfcb" Nov 03 20:35:29.560905 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:29.560875 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1c4a0757d3567e9ac47634d122b35a2841961058d97410a4e2a6ebd76c2cfcb\": container with ID starting with b1c4a0757d3567e9ac47634d122b35a2841961058d97410a4e2a6ebd76c2cfcb not found: ID does not exist" containerID="b1c4a0757d3567e9ac47634d122b35a2841961058d97410a4e2a6ebd76c2cfcb" Nov 03 20:35:29.561007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.560917 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1c4a0757d3567e9ac47634d122b35a2841961058d97410a4e2a6ebd76c2cfcb"} err="failed to get container status \"b1c4a0757d3567e9ac47634d122b35a2841961058d97410a4e2a6ebd76c2cfcb\": rpc error: code = NotFound desc = could not find container \"b1c4a0757d3567e9ac47634d122b35a2841961058d97410a4e2a6ebd76c2cfcb\": container with ID starting with b1c4a0757d3567e9ac47634d122b35a2841961058d97410a4e2a6ebd76c2cfcb not found: ID does not exist" Nov 03 20:35:29.566774 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.566745 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g"] Nov 03 20:35:29.570076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.570050 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-3aba31-6b84b4698c-4jf8g"] Nov 03 20:35:29.572767 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.572737 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv25h\" (UniqueName: \"kubernetes.io/projected/250b873d-632c-4169-b357-77f6d8b9e9b6-kube-api-access-bv25h\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:29.572872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.572771 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250b873d-632c-4169-b357-77f6d8b9e9b6-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:29.585743 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.582344 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l"] Nov 03 20:35:29.595493 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.595464 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-3aba31-86dbd748b8-pjl8l"] Nov 03 20:35:29.758475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.758433 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250b873d-632c-4169-b357-77f6d8b9e9b6" path="/var/lib/kubelet/pods/250b873d-632c-4169-b357-77f6d8b9e9b6/volumes" Nov 03 20:35:29.759007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:29.758990 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de4b055-a669-4760-ba03-041291c506fd" path="/var/lib/kubelet/pods/4de4b055-a669-4760-ba03-041291c506fd/volumes" Nov 03 20:35:31.569908 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.569874 4457 generic.go:356] "Generic (PLEG): container finished" podID="6cf12593-ef2a-4848-be72-ce57d759c2e9" containerID="b8049778edcb9e1b90bdca5cb8dcf3f227004ba34cc059918b1ed4a6b7e23b32" exitCode=0 Nov 03 20:35:31.570336 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.569951 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" event={"ID":"6cf12593-ef2a-4848-be72-ce57d759c2e9","Type":"ContainerDied","Data":"b8049778edcb9e1b90bdca5cb8dcf3f227004ba34cc059918b1ed4a6b7e23b32"} Nov 03 20:35:31.620465 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.620402 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:35:31.694898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.694863 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-envoy\") pod \"6cf12593-ef2a-4848-be72-ce57d759c2e9\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " Nov 03 20:35:31.694898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.694904 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cf12593-ef2a-4848-be72-ce57d759c2e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf12593-ef2a-4848-be72-ce57d759c2e9-istiod-ca-cert\") pod \"6cf12593-ef2a-4848-be72-ce57d759c2e9\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " Nov 03 20:35:31.695157 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.694929 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-data\") pod \"6cf12593-ef2a-4848-be72-ce57d759c2e9\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " Nov 03 20:35:31.695157 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.694997 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-credential-socket\") pod \"6cf12593-ef2a-4848-be72-ce57d759c2e9\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " Nov 03 20:35:31.695157 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.695037 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-token\" (UniqueName: \"kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-token\") pod \"6cf12593-ef2a-4848-be72-ce57d759c2e9\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " Nov 03 20:35:31.695157 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.695088 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-podinfo\") pod \"6cf12593-ef2a-4848-be72-ce57d759c2e9\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " Nov 03 20:35:31.695157 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.695117 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-kube-api-access-p5v78\" (UniqueName: \"kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-kube-api-access-p5v78\") pod \"6cf12593-ef2a-4848-be72-ce57d759c2e9\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " Nov 03 20:35:31.695478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.695164 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-socket\") pod \"6cf12593-ef2a-4848-be72-ce57d759c2e9\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " Nov 03 20:35:31.695478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.695217 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-certs\") pod \"6cf12593-ef2a-4848-be72-ce57d759c2e9\" (UID: \"6cf12593-ef2a-4848-be72-ce57d759c2e9\") " Nov 03 20:35:31.695478 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.695334 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-data" pod "6cf12593-ef2a-4848-be72-ce57d759c2e9" (UID: "6cf12593-ef2a-4848-be72-ce57d759c2e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.695651 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.695580 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-socket" pod "6cf12593-ef2a-4848-be72-ce57d759c2e9" (UID: "6cf12593-ef2a-4848-be72-ce57d759c2e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.695706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.695663 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf12593-ef2a-4848-be72-ce57d759c2e9-istiod-ca-cert" pod "6cf12593-ef2a-4848-be72-ce57d759c2e9" (UID: "6cf12593-ef2a-4848-be72-ce57d759c2e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:31.695706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.695684 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-credential-socket" pod "6cf12593-ef2a-4848-be72-ce57d759c2e9" (UID: "6cf12593-ef2a-4848-be72-ce57d759c2e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.695823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.695735 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.695823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.695753 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.695823 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.695788 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-certs" pod "6cf12593-ef2a-4848-be72-ce57d759c2e9" (UID: "6cf12593-ef2a-4848-be72-ce57d759c2e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.697352 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.697327 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-envoy" pod "6cf12593-ef2a-4848-be72-ce57d759c2e9" (UID: "6cf12593-ef2a-4848-be72-ce57d759c2e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:31.697504 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.697486 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-token" pod "6cf12593-ef2a-4848-be72-ce57d759c2e9" (UID: "6cf12593-ef2a-4848-be72-ce57d759c2e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:31.697623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.697589 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-podinfo" pod "6cf12593-ef2a-4848-be72-ce57d759c2e9" (UID: "6cf12593-ef2a-4848-be72-ce57d759c2e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:31.697683 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.697616 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-kube-api-access-p5v78" pod "6cf12593-ef2a-4848-be72-ce57d759c2e9" (UID: "6cf12593-ef2a-4848-be72-ce57d759c2e9"). InnerVolumeSpecName "kube-api-access-p5v78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:31.796636 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.796587 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cf12593-ef2a-4848-be72-ce57d759c2e9-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.796636 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.796639 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.796878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.796656 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.796878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.796671 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.796878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.796686 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5v78\" (UniqueName: \"kubernetes.io/projected/6cf12593-ef2a-4848-be72-ce57d759c2e9-kube-api-access-p5v78\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.796878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.796710 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:31.796878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:31.796725 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cf12593-ef2a-4848-be72-ce57d759c2e9-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:32.580154 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:32.580122 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" Nov 03 20:35:32.580605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:32.580135 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh" event={"ID":"6cf12593-ef2a-4848-be72-ce57d759c2e9","Type":"ContainerDied","Data":"4add2406c38992eaff3bd74ce8fcc867b6fc71515992fdd3bf8e8c17b40dcf51"} Nov 03 20:35:32.580605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:32.580224 4457 scope.go:122] "RemoveContainer" containerID="b8049778edcb9e1b90bdca5cb8dcf3f227004ba34cc059918b1ed4a6b7e23b32" Nov 03 20:35:32.600851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:32.600826 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh"] Nov 03 20:35:32.608593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:32.608571 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d412cf-openshift-default-76ddb4c6f7-9g5vh"] Nov 03 20:35:33.335453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.335425 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:35:33.413541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.413462 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-token\" (UniqueName: \"kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-token\") pod \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " Nov 03 20:35:33.413541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.413509 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-envoy\") pod \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " Nov 03 20:35:33.413778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.413573 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-credential-socket\") pod \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " Nov 03 20:35:33.413778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.413601 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istiod-ca-cert\") pod \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " Nov 03 20:35:33.413778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.413654 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-kube-api-access-f6kcv\" (UniqueName: \"kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-kube-api-access-f6kcv\") pod \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " Nov 03 20:35:33.413778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.413708 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-socket\") pod \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " Nov 03 20:35:33.413778 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.413742 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-podinfo\") pod \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " Nov 03 20:35:33.414074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.413807 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-certs\") pod \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " Nov 03 20:35:33.414074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.413848 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-data\") pod \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\" (UID: \"97f3184c-7322-4684-94e1-9e5e4eecd6e7\") " Nov 03 20:35:33.414074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.413986 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-socket" pod "97f3184c-7322-4684-94e1-9e5e4eecd6e7" (UID: "97f3184c-7322-4684-94e1-9e5e4eecd6e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.414249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.414197 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-credential-socket" pod "97f3184c-7322-4684-94e1-9e5e4eecd6e7" (UID: "97f3184c-7322-4684-94e1-9e5e4eecd6e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.414249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.414212 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-data" pod "97f3184c-7322-4684-94e1-9e5e4eecd6e7" (UID: "97f3184c-7322-4684-94e1-9e5e4eecd6e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.414379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.414340 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.414379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.414353 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.414379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.414363 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.414379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.414371 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-certs" pod "97f3184c-7322-4684-94e1-9e5e4eecd6e7" (UID: "97f3184c-7322-4684-94e1-9e5e4eecd6e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.414632 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.414448 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istiod-ca-cert" pod "97f3184c-7322-4684-94e1-9e5e4eecd6e7" (UID: "97f3184c-7322-4684-94e1-9e5e4eecd6e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:33.416140 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.416111 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-token" pod "97f3184c-7322-4684-94e1-9e5e4eecd6e7" (UID: "97f3184c-7322-4684-94e1-9e5e4eecd6e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:33.416248 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.416142 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-envoy" pod "97f3184c-7322-4684-94e1-9e5e4eecd6e7" (UID: "97f3184c-7322-4684-94e1-9e5e4eecd6e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:33.416300 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.416242 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-podinfo" pod "97f3184c-7322-4684-94e1-9e5e4eecd6e7" (UID: "97f3184c-7322-4684-94e1-9e5e4eecd6e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:33.416501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.416479 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-kube-api-access-f6kcv" pod "97f3184c-7322-4684-94e1-9e5e4eecd6e7" (UID: "97f3184c-7322-4684-94e1-9e5e4eecd6e7"). InnerVolumeSpecName "kube-api-access-f6kcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:33.515648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.515608 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.515648 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.515644 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.515855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.515659 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.515855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.515671 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.515855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.515683 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97f3184c-7322-4684-94e1-9e5e4eecd6e7-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.515855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.515696 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6kcv\" (UniqueName: \"kubernetes.io/projected/97f3184c-7322-4684-94e1-9e5e4eecd6e7-kube-api-access-f6kcv\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:33.589402 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.589359 4457 generic.go:356] "Generic (PLEG): container finished" podID="97f3184c-7322-4684-94e1-9e5e4eecd6e7" containerID="de1a8c74891e1f40121ffa0dbba05bc979f4d253660c1b97c7bfcd3b45a80b2c" exitCode=0 Nov 03 20:35:33.589893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.589444 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" Nov 03 20:35:33.589893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.589519 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" event={"ID":"97f3184c-7322-4684-94e1-9e5e4eecd6e7","Type":"ContainerDied","Data":"de1a8c74891e1f40121ffa0dbba05bc979f4d253660c1b97c7bfcd3b45a80b2c"} Nov 03 20:35:33.589893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.589600 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf" event={"ID":"97f3184c-7322-4684-94e1-9e5e4eecd6e7","Type":"ContainerDied","Data":"d0ff136a0c4617f7c6ebdfa46eaf2c781020c5d3ad07e39bdc28a270121b2029"} Nov 03 20:35:33.589893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.589633 4457 scope.go:122] "RemoveContainer" containerID="de1a8c74891e1f40121ffa0dbba05bc979f4d253660c1b97c7bfcd3b45a80b2c" Nov 03 20:35:33.601377 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.601362 4457 scope.go:122] "RemoveContainer" containerID="de1a8c74891e1f40121ffa0dbba05bc979f4d253660c1b97c7bfcd3b45a80b2c" Nov 03 20:35:33.601689 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:33.601668 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1a8c74891e1f40121ffa0dbba05bc979f4d253660c1b97c7bfcd3b45a80b2c\": container with ID starting with de1a8c74891e1f40121ffa0dbba05bc979f4d253660c1b97c7bfcd3b45a80b2c not found: ID does not exist" containerID="de1a8c74891e1f40121ffa0dbba05bc979f4d253660c1b97c7bfcd3b45a80b2c" Nov 03 20:35:33.601741 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.601696 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1a8c74891e1f40121ffa0dbba05bc979f4d253660c1b97c7bfcd3b45a80b2c"} err="failed to get container status \"de1a8c74891e1f40121ffa0dbba05bc979f4d253660c1b97c7bfcd3b45a80b2c\": rpc error: code = NotFound desc = could not find container \"de1a8c74891e1f40121ffa0dbba05bc979f4d253660c1b97c7bfcd3b45a80b2c\": container with ID starting with de1a8c74891e1f40121ffa0dbba05bc979f4d253660c1b97c7bfcd3b45a80b2c not found: ID does not exist" Nov 03 20:35:33.614549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.614520 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf"] Nov 03 20:35:33.622260 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.622236 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-3aba31-openshift-default-5f6d888c6c-djmkf"] Nov 03 20:35:33.757638 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.757562 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf12593-ef2a-4848-be72-ce57d759c2e9" path="/var/lib/kubelet/pods/6cf12593-ef2a-4848-be72-ce57d759c2e9/volumes" Nov 03 20:35:33.758017 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:33.758003 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f3184c-7322-4684-94e1-9e5e4eecd6e7" path="/var/lib/kubelet/pods/97f3184c-7322-4684-94e1-9e5e4eecd6e7/volumes" Nov 03 20:35:34.091191 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:34.091133 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" podUID="37cbe749-2e98-4766-b527-8aa40a28e2b2" containerName="payload-pre-processing" containerID="cri-o://b31f021a3ae05450b2ab3fc5bd28d1643ed1e0ea199c21739bc5174498435547" gracePeriod=25 Nov 03 20:35:35.999492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:35.999448 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" podUID="37cbe749-2e98-4766-b527-8aa40a28e2b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:35:36.151157 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:36.151120 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:35:38.930406 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:38.930355 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7"] Nov 03 20:35:38.930905 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:38.930622 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" containerID="cri-o://086b31bb286d6ace17871f003864f076fa7e04033a60988e6d5bb2fd8a240ab8" gracePeriod=30 Nov 03 20:35:44.222172 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.222042 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:35:44.328984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.328944 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-socket\") pod \"f14a467c-de34-4c09-837d-ef5fb99986b7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " Nov 03 20:35:44.329190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329007 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-podinfo\") pod \"f14a467c-de34-4c09-837d-ef5fb99986b7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " Nov 03 20:35:44.329190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329040 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-data\") pod \"f14a467c-de34-4c09-837d-ef5fb99986b7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " Nov 03 20:35:44.329190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329098 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-envoy\") pod \"f14a467c-de34-4c09-837d-ef5fb99986b7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " Nov 03 20:35:44.329190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329130 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-kube-api-access-jpw4h\" (UniqueName: \"kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-kube-api-access-jpw4h\") pod \"f14a467c-de34-4c09-837d-ef5fb99986b7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " Nov 03 20:35:44.329190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329157 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f14a467c-de34-4c09-837d-ef5fb99986b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14a467c-de34-4c09-837d-ef5fb99986b7-istiod-ca-cert\") pod \"f14a467c-de34-4c09-837d-ef5fb99986b7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " Nov 03 20:35:44.329527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329270 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-socket" pod "f14a467c-de34-4c09-837d-ef5fb99986b7" (UID: "f14a467c-de34-4c09-837d-ef5fb99986b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.329527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329278 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-credential-socket\") pod \"f14a467c-de34-4c09-837d-ef5fb99986b7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " Nov 03 20:35:44.329527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329383 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-data" pod "f14a467c-de34-4c09-837d-ef5fb99986b7" (UID: "f14a467c-de34-4c09-837d-ef5fb99986b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.329527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329425 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-certs\") pod \"f14a467c-de34-4c09-837d-ef5fb99986b7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " Nov 03 20:35:44.329527 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329483 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-token\" (UniqueName: \"kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-token\") pod \"f14a467c-de34-4c09-837d-ef5fb99986b7\" (UID: \"f14a467c-de34-4c09-837d-ef5fb99986b7\") " Nov 03 20:35:44.329756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329544 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-credential-socket" pod "f14a467c-de34-4c09-837d-ef5fb99986b7" (UID: "f14a467c-de34-4c09-837d-ef5fb99986b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.329756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329702 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-certs" pod "f14a467c-de34-4c09-837d-ef5fb99986b7" (UID: "f14a467c-de34-4c09-837d-ef5fb99986b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.329756 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329635 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14a467c-de34-4c09-837d-ef5fb99986b7-istiod-ca-cert" pod "f14a467c-de34-4c09-837d-ef5fb99986b7" (UID: "f14a467c-de34-4c09-837d-ef5fb99986b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:44.329979 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329956 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14a467c-de34-4c09-837d-ef5fb99986b7-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.330050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329983 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.330050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.329994 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.330050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.330003 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.330050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.330012 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.331492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.331453 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-podinfo" pod "f14a467c-de34-4c09-837d-ef5fb99986b7" (UID: "f14a467c-de34-4c09-837d-ef5fb99986b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:35:44.331986 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.331937 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-token" pod "f14a467c-de34-4c09-837d-ef5fb99986b7" (UID: "f14a467c-de34-4c09-837d-ef5fb99986b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:44.332266 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.332237 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-kube-api-access-jpw4h" pod "f14a467c-de34-4c09-837d-ef5fb99986b7" (UID: "f14a467c-de34-4c09-837d-ef5fb99986b7"). InnerVolumeSpecName "kube-api-access-jpw4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:44.332266 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.332256 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-envoy" pod "f14a467c-de34-4c09-837d-ef5fb99986b7" (UID: "f14a467c-de34-4c09-837d-ef5fb99986b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:35:44.431022 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.430981 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.431022 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.431020 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.431688 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.431034 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpw4h\" (UniqueName: \"kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-kube-api-access-jpw4h\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.431688 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.431079 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14a467c-de34-4c09-837d-ef5fb99986b7-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:44.698249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.698213 4457 generic.go:356] "Generic (PLEG): container finished" podID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerID="086b31bb286d6ace17871f003864f076fa7e04033a60988e6d5bb2fd8a240ab8" exitCode=0 Nov 03 20:35:44.698469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.698279 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" Nov 03 20:35:44.698469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.698298 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" event={"ID":"f14a467c-de34-4c09-837d-ef5fb99986b7","Type":"ContainerDied","Data":"086b31bb286d6ace17871f003864f076fa7e04033a60988e6d5bb2fd8a240ab8"} Nov 03 20:35:44.698469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.698364 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7" event={"ID":"f14a467c-de34-4c09-837d-ef5fb99986b7","Type":"ContainerDied","Data":"00507a996046ff2f2ac4d30b96db6810ef4b7e4f1d2fab2bb997cf06d044511d"} Nov 03 20:35:44.698469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.698405 4457 scope.go:122] "RemoveContainer" containerID="086b31bb286d6ace17871f003864f076fa7e04033a60988e6d5bb2fd8a240ab8" Nov 03 20:35:44.710383 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.710362 4457 scope.go:122] "RemoveContainer" containerID="086b31bb286d6ace17871f003864f076fa7e04033a60988e6d5bb2fd8a240ab8" Nov 03 20:35:44.710699 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:44.710676 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"086b31bb286d6ace17871f003864f076fa7e04033a60988e6d5bb2fd8a240ab8\": container with ID starting with 086b31bb286d6ace17871f003864f076fa7e04033a60988e6d5bb2fd8a240ab8 not found: ID does not exist" containerID="086b31bb286d6ace17871f003864f076fa7e04033a60988e6d5bb2fd8a240ab8" Nov 03 20:35:44.710804 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.710706 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086b31bb286d6ace17871f003864f076fa7e04033a60988e6d5bb2fd8a240ab8"} err="failed to get container status \"086b31bb286d6ace17871f003864f076fa7e04033a60988e6d5bb2fd8a240ab8\": rpc error: code = NotFound desc = could not find container \"086b31bb286d6ace17871f003864f076fa7e04033a60988e6d5bb2fd8a240ab8\": container with ID starting with 086b31bb286d6ace17871f003864f076fa7e04033a60988e6d5bb2fd8a240ab8 not found: ID does not exist" Nov 03 20:35:44.726234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.726203 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7"] Nov 03 20:35:44.730497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:44.730475 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a6f98b-openshift-default-79fb5bf5f-6g9g7"] Nov 03 20:35:45.756426 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:45.756371 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" path="/var/lib/kubelet/pods/f14a467c-de34-4c09-837d-ef5fb99986b7/volumes" Nov 03 20:35:45.999066 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:45.999021 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" podUID="37cbe749-2e98-4766-b527-8aa40a28e2b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:35:54.337438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.337379 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" Nov 03 20:35:54.440519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.440440 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37cbe749-2e98-4766-b527-8aa40a28e2b2-kube-api-access-jdbkx\" (UniqueName: \"kubernetes.io/projected/37cbe749-2e98-4766-b527-8aa40a28e2b2-kube-api-access-jdbkx\") pod \"37cbe749-2e98-4766-b527-8aa40a28e2b2\" (UID: \"37cbe749-2e98-4766-b527-8aa40a28e2b2\") " Nov 03 20:35:54.440519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.440510 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37cbe749-2e98-4766-b527-8aa40a28e2b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cbe749-2e98-4766-b527-8aa40a28e2b2-plugins-config-volume\") pod \"37cbe749-2e98-4766-b527-8aa40a28e2b2\" (UID: \"37cbe749-2e98-4766-b527-8aa40a28e2b2\") " Nov 03 20:35:54.440858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.440836 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cbe749-2e98-4766-b527-8aa40a28e2b2-plugins-config-volume" pod "37cbe749-2e98-4766-b527-8aa40a28e2b2" (UID: "37cbe749-2e98-4766-b527-8aa40a28e2b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:54.442581 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.442556 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cbe749-2e98-4766-b527-8aa40a28e2b2-kube-api-access-jdbkx" pod "37cbe749-2e98-4766-b527-8aa40a28e2b2" (UID: "37cbe749-2e98-4766-b527-8aa40a28e2b2"). InnerVolumeSpecName "kube-api-access-jdbkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:54.541838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.541799 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37cbe749-2e98-4766-b527-8aa40a28e2b2-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:54.541838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.541831 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdbkx\" (UniqueName: \"kubernetes.io/projected/37cbe749-2e98-4766-b527-8aa40a28e2b2-kube-api-access-jdbkx\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:54.779350 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.779266 4457 generic.go:356] "Generic (PLEG): container finished" podID="37cbe749-2e98-4766-b527-8aa40a28e2b2" containerID="b31f021a3ae05450b2ab3fc5bd28d1643ed1e0ea199c21739bc5174498435547" exitCode=0 Nov 03 20:35:54.779350 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.779308 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" event={"ID":"37cbe749-2e98-4766-b527-8aa40a28e2b2","Type":"ContainerDied","Data":"b31f021a3ae05450b2ab3fc5bd28d1643ed1e0ea199c21739bc5174498435547"} Nov 03 20:35:54.779350 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.779319 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" Nov 03 20:35:54.779350 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.779343 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl" event={"ID":"37cbe749-2e98-4766-b527-8aa40a28e2b2","Type":"ContainerDied","Data":"3aafdcd282c793e7205f8847a70cf21cb58869a759cb216b11bf72a048230e14"} Nov 03 20:35:54.779708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.779365 4457 scope.go:122] "RemoveContainer" containerID="b31f021a3ae05450b2ab3fc5bd28d1643ed1e0ea199c21739bc5174498435547" Nov 03 20:35:54.790241 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.790217 4457 scope.go:122] "RemoveContainer" containerID="b31f021a3ae05450b2ab3fc5bd28d1643ed1e0ea199c21739bc5174498435547" Nov 03 20:35:54.790558 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:54.790536 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31f021a3ae05450b2ab3fc5bd28d1643ed1e0ea199c21739bc5174498435547\": container with ID starting with b31f021a3ae05450b2ab3fc5bd28d1643ed1e0ea199c21739bc5174498435547 not found: ID does not exist" containerID="b31f021a3ae05450b2ab3fc5bd28d1643ed1e0ea199c21739bc5174498435547" Nov 03 20:35:54.790605 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.790567 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31f021a3ae05450b2ab3fc5bd28d1643ed1e0ea199c21739bc5174498435547"} err="failed to get container status \"b31f021a3ae05450b2ab3fc5bd28d1643ed1e0ea199c21739bc5174498435547\": rpc error: code = NotFound desc = could not find container \"b31f021a3ae05450b2ab3fc5bd28d1643ed1e0ea199c21739bc5174498435547\": container with ID starting with b31f021a3ae05450b2ab3fc5bd28d1643ed1e0ea199c21739bc5174498435547 not found: ID does not exist" Nov 03 20:35:54.804112 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.804087 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl"] Nov 03 20:35:54.808430 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:54.808407 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-45e569-f67b4cc8c-64dtl"] Nov 03 20:35:55.756242 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:55.756206 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37cbe749-2e98-4766-b527-8aa40a28e2b2" path="/var/lib/kubelet/pods/37cbe749-2e98-4766-b527-8aa40a28e2b2/volumes" Nov 03 20:35:57.723529 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:57.723493 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx"] Nov 03 20:35:57.739311 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:57.739278 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr"] Nov 03 20:35:58.037408 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.037314 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh"] Nov 03 20:35:58.038353 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.038137 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" podUID="c67ccec4-726a-4e35-a85f-baf6ff097782" containerName="maas-api" containerID="cri-o://936fa7685dc62b50e0c192fb5c651a916c49bbcdfbb6e280ea6ec9398d568873" gracePeriod=30 Nov 03 20:35:58.335930 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.335903 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:35:58.481035 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.481003 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-maas-api-tls\") pod \"c67ccec4-726a-4e35-a85f-baf6ff097782\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " Nov 03 20:35:58.481035 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.481043 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-metrics-certs\") pod \"c67ccec4-726a-4e35-a85f-baf6ff097782\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " Nov 03 20:35:58.481317 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.481092 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67ccec4-726a-4e35-a85f-baf6ff097782-kube-api-access-ckzf9\" (UniqueName: \"kubernetes.io/projected/c67ccec4-726a-4e35-a85f-baf6ff097782-kube-api-access-ckzf9\") pod \"c67ccec4-726a-4e35-a85f-baf6ff097782\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " Nov 03 20:35:58.481317 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.481191 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c67ccec4-726a-4e35-a85f-baf6ff097782-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67ccec4-726a-4e35-a85f-baf6ff097782-openshift-service-ca\") pod \"c67ccec4-726a-4e35-a85f-baf6ff097782\" (UID: \"c67ccec4-726a-4e35-a85f-baf6ff097782\") " Nov 03 20:35:58.481907 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.481872 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67ccec4-726a-4e35-a85f-baf6ff097782-openshift-service-ca" pod "c67ccec4-726a-4e35-a85f-baf6ff097782" (UID: "c67ccec4-726a-4e35-a85f-baf6ff097782"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:35:58.484018 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.483990 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-maas-api-tls" pod "c67ccec4-726a-4e35-a85f-baf6ff097782" (UID: "c67ccec4-726a-4e35-a85f-baf6ff097782"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:58.484116 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.484008 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67ccec4-726a-4e35-a85f-baf6ff097782-kube-api-access-ckzf9" pod "c67ccec4-726a-4e35-a85f-baf6ff097782" (UID: "c67ccec4-726a-4e35-a85f-baf6ff097782"). InnerVolumeSpecName "kube-api-access-ckzf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:35:58.484171 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.484144 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-metrics-certs" pod "c67ccec4-726a-4e35-a85f-baf6ff097782" (UID: "c67ccec4-726a-4e35-a85f-baf6ff097782"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:35:58.582790 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.582703 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-maas-api-tls\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:58.582790 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.582737 4457 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67ccec4-726a-4e35-a85f-baf6ff097782-metrics-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:58.582790 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.582770 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckzf9\" (UniqueName: \"kubernetes.io/projected/c67ccec4-726a-4e35-a85f-baf6ff097782-kube-api-access-ckzf9\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:58.582790 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.582782 4457 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67ccec4-726a-4e35-a85f-baf6ff097782-openshift-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:35:58.813752 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.813718 4457 generic.go:356] "Generic (PLEG): container finished" podID="c67ccec4-726a-4e35-a85f-baf6ff097782" containerID="936fa7685dc62b50e0c192fb5c651a916c49bbcdfbb6e280ea6ec9398d568873" exitCode=0 Nov 03 20:35:58.814261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.813774 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" Nov 03 20:35:58.814261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.813804 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" event={"ID":"c67ccec4-726a-4e35-a85f-baf6ff097782","Type":"ContainerDied","Data":"936fa7685dc62b50e0c192fb5c651a916c49bbcdfbb6e280ea6ec9398d568873"} Nov 03 20:35:58.814261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.813860 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh" event={"ID":"c67ccec4-726a-4e35-a85f-baf6ff097782","Type":"ContainerDied","Data":"fd995ea5751ebdd97319c6294e70973f3679434d73fe9f1ae546cbb1a7d613e6"} Nov 03 20:35:58.814261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.813891 4457 scope.go:122] "RemoveContainer" containerID="936fa7685dc62b50e0c192fb5c651a916c49bbcdfbb6e280ea6ec9398d568873" Nov 03 20:35:58.825484 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.825379 4457 scope.go:122] "RemoveContainer" containerID="936fa7685dc62b50e0c192fb5c651a916c49bbcdfbb6e280ea6ec9398d568873" Nov 03 20:35:58.825787 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:35:58.825766 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936fa7685dc62b50e0c192fb5c651a916c49bbcdfbb6e280ea6ec9398d568873\": container with ID starting with 936fa7685dc62b50e0c192fb5c651a916c49bbcdfbb6e280ea6ec9398d568873 not found: ID does not exist" containerID="936fa7685dc62b50e0c192fb5c651a916c49bbcdfbb6e280ea6ec9398d568873" Nov 03 20:35:58.825872 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.825800 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936fa7685dc62b50e0c192fb5c651a916c49bbcdfbb6e280ea6ec9398d568873"} err="failed to get container status \"936fa7685dc62b50e0c192fb5c651a916c49bbcdfbb6e280ea6ec9398d568873\": rpc error: code = NotFound desc = could not find container \"936fa7685dc62b50e0c192fb5c651a916c49bbcdfbb6e280ea6ec9398d568873\": container with ID starting with 936fa7685dc62b50e0c192fb5c651a916c49bbcdfbb6e280ea6ec9398d568873 not found: ID does not exist" Nov 03 20:35:58.838477 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.838423 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh"] Nov 03 20:35:58.842591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:58.842570 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c60d2a-866f6f4675-hrhbh"] Nov 03 20:35:59.756690 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:35:59.756650 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c67ccec4-726a-4e35-a85f-baf6ff097782" path="/var/lib/kubelet/pods/c67ccec4-726a-4e35-a85f-baf6ff097782/volumes" Nov 03 20:36:02.723887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:02.723798 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" podUID="32bb3d1e-c3a2-401a-be41-dc09cebde931" containerName="payload-processing" containerID="cri-o://5c41416f32d196011791f63532c18ba3301ba31f14a12deb769d1414af6ed21c" gracePeriod=25 Nov 03 20:36:02.739782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:02.739721 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" podUID="fdc31109-670b-421b-8d51-5bb50801cb55" containerName="payload-pre-processing" containerID="cri-o://3a773e6e04195e4c164086b1189bdf6d02acbd500778b02a017c36f41043a74c" gracePeriod=25 Nov 03 20:36:02.851790 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:02.851763 4457 generic.go:356] "Generic (PLEG): container finished" podID="32bb3d1e-c3a2-401a-be41-dc09cebde931" containerID="5c41416f32d196011791f63532c18ba3301ba31f14a12deb769d1414af6ed21c" exitCode=0 Nov 03 20:36:02.851910 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:02.851841 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" event={"ID":"32bb3d1e-c3a2-401a-be41-dc09cebde931","Type":"ContainerDied","Data":"5c41416f32d196011791f63532c18ba3301ba31f14a12deb769d1414af6ed21c"} Nov 03 20:36:02.989930 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:02.989862 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" Nov 03 20:36:03.130848 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:03.130812 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32bb3d1e-c3a2-401a-be41-dc09cebde931-kube-api-access-x98wj\" (UniqueName: \"kubernetes.io/projected/32bb3d1e-c3a2-401a-be41-dc09cebde931-kube-api-access-x98wj\") pod \"32bb3d1e-c3a2-401a-be41-dc09cebde931\" (UID: \"32bb3d1e-c3a2-401a-be41-dc09cebde931\") " Nov 03 20:36:03.131004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:03.130903 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32bb3d1e-c3a2-401a-be41-dc09cebde931-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32bb3d1e-c3a2-401a-be41-dc09cebde931-plugins-config-volume\") pod \"32bb3d1e-c3a2-401a-be41-dc09cebde931\" (UID: \"32bb3d1e-c3a2-401a-be41-dc09cebde931\") " Nov 03 20:36:03.131295 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:03.131265 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32bb3d1e-c3a2-401a-be41-dc09cebde931-plugins-config-volume" pod "32bb3d1e-c3a2-401a-be41-dc09cebde931" (UID: "32bb3d1e-c3a2-401a-be41-dc09cebde931"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:03.133022 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:03.133002 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32bb3d1e-c3a2-401a-be41-dc09cebde931-kube-api-access-x98wj" pod "32bb3d1e-c3a2-401a-be41-dc09cebde931" (UID: "32bb3d1e-c3a2-401a-be41-dc09cebde931"). InnerVolumeSpecName "kube-api-access-x98wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:03.231789 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:03.231748 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x98wj\" (UniqueName: \"kubernetes.io/projected/32bb3d1e-c3a2-401a-be41-dc09cebde931-kube-api-access-x98wj\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:03.231789 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:03.231777 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32bb3d1e-c3a2-401a-be41-dc09cebde931-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:03.860364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:03.860330 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" Nov 03 20:36:03.860364 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:03.860350 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx" event={"ID":"32bb3d1e-c3a2-401a-be41-dc09cebde931","Type":"ContainerDied","Data":"37d0f1693ddf08a693f83764e4539d67c3d5685b13154161fdd75adb7ce55356"} Nov 03 20:36:03.860933 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:03.860453 4457 scope.go:122] "RemoveContainer" containerID="5c41416f32d196011791f63532c18ba3301ba31f14a12deb769d1414af6ed21c" Nov 03 20:36:03.882532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:03.882500 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx"] Nov 03 20:36:03.887049 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:03.887024 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c60d2a-86c5ff854-nkmxx"] Nov 03 20:36:05.758658 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:05.758616 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32bb3d1e-c3a2-401a-be41-dc09cebde931" path="/var/lib/kubelet/pods/32bb3d1e-c3a2-401a-be41-dc09cebde931/volumes" Nov 03 20:36:06.146798 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:36:06.146759 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:36:08.083548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:08.083515 4457 scope.go:122] "RemoveContainer" containerID="575f3c1efe760867cdd55d210d287eafe57b007c2cada8e35fa2cc06b7ca6026" Nov 03 20:36:08.093421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:08.093377 4457 scope.go:122] "RemoveContainer" containerID="bb1a40645b3a4a7a180c45273c1cfd6d697818af9e4c6c313f58193a09fcfcdb" Nov 03 20:36:11.254417 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:11.254370 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc"] Nov 03 20:36:11.284193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:11.284162 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm"] Nov 03 20:36:12.430987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:12.430947 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" podUID="fdc31109-670b-421b-8d51-5bb50801cb55" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:36:16.617783 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:16.617745 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4"] Nov 03 20:36:21.618814 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:21.618765 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" podUID="54c875b5-372c-42fe-828a-7605988b595d" containerName="payload-pre-processing" containerID="cri-o://fe80e7cabbbff7b9e436b7e98a7b620a4b1da0df852dfb8836c8fb7c09258cfb" gracePeriod=25 Nov 03 20:36:21.962525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:21.962438 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" podUID="54c875b5-372c-42fe-828a-7605988b595d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:36:22.431575 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:22.431538 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" podUID="fdc31109-670b-421b-8d51-5bb50801cb55" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:36:22.997205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:22.997129 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" Nov 03 20:36:23.004677 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.004648 4457 generic.go:356] "Generic (PLEG): container finished" podID="fdc31109-670b-421b-8d51-5bb50801cb55" containerID="3a773e6e04195e4c164086b1189bdf6d02acbd500778b02a017c36f41043a74c" exitCode=0 Nov 03 20:36:23.004815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.004685 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" Nov 03 20:36:23.004815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.004707 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" event={"ID":"fdc31109-670b-421b-8d51-5bb50801cb55","Type":"ContainerDied","Data":"3a773e6e04195e4c164086b1189bdf6d02acbd500778b02a017c36f41043a74c"} Nov 03 20:36:23.004815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.004752 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr" event={"ID":"fdc31109-670b-421b-8d51-5bb50801cb55","Type":"ContainerDied","Data":"d71fb928077b8c69e6f3e9a6b8bc997361868da88de8df2652855897b7879f83"} Nov 03 20:36:23.004815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.004775 4457 scope.go:122] "RemoveContainer" containerID="3a773e6e04195e4c164086b1189bdf6d02acbd500778b02a017c36f41043a74c" Nov 03 20:36:23.014861 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.014841 4457 scope.go:122] "RemoveContainer" containerID="3a773e6e04195e4c164086b1189bdf6d02acbd500778b02a017c36f41043a74c" Nov 03 20:36:23.015266 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:36:23.015232 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a773e6e04195e4c164086b1189bdf6d02acbd500778b02a017c36f41043a74c\": container with ID starting with 3a773e6e04195e4c164086b1189bdf6d02acbd500778b02a017c36f41043a74c not found: ID does not exist" containerID="3a773e6e04195e4c164086b1189bdf6d02acbd500778b02a017c36f41043a74c" Nov 03 20:36:23.015398 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.015264 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a773e6e04195e4c164086b1189bdf6d02acbd500778b02a017c36f41043a74c"} err="failed to get container status \"3a773e6e04195e4c164086b1189bdf6d02acbd500778b02a017c36f41043a74c\": rpc error: code = NotFound desc = could not find container \"3a773e6e04195e4c164086b1189bdf6d02acbd500778b02a017c36f41043a74c\": container with ID starting with 3a773e6e04195e4c164086b1189bdf6d02acbd500778b02a017c36f41043a74c not found: ID does not exist" Nov 03 20:36:23.028487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.028449 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh"] Nov 03 20:36:23.029413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.029362 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-861df19a-2mqzh"] Nov 03 20:36:23.147203 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.147167 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc31109-670b-421b-8d51-5bb50801cb55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc31109-670b-421b-8d51-5bb50801cb55-plugins-config-volume\") pod \"fdc31109-670b-421b-8d51-5bb50801cb55\" (UID: \"fdc31109-670b-421b-8d51-5bb50801cb55\") " Nov 03 20:36:23.147374 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.147300 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc31109-670b-421b-8d51-5bb50801cb55-kube-api-access-lctvl\" (UniqueName: \"kubernetes.io/projected/fdc31109-670b-421b-8d51-5bb50801cb55-kube-api-access-lctvl\") pod \"fdc31109-670b-421b-8d51-5bb50801cb55\" (UID: \"fdc31109-670b-421b-8d51-5bb50801cb55\") " Nov 03 20:36:23.147587 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.147565 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc31109-670b-421b-8d51-5bb50801cb55-plugins-config-volume" pod "fdc31109-670b-421b-8d51-5bb50801cb55" (UID: "fdc31109-670b-421b-8d51-5bb50801cb55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:23.149309 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.149286 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc31109-670b-421b-8d51-5bb50801cb55-kube-api-access-lctvl" pod "fdc31109-670b-421b-8d51-5bb50801cb55" (UID: "fdc31109-670b-421b-8d51-5bb50801cb55"). InnerVolumeSpecName "kube-api-access-lctvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:23.248353 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.248267 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdc31109-670b-421b-8d51-5bb50801cb55-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:23.248353 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.248299 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lctvl\" (UniqueName: \"kubernetes.io/projected/fdc31109-670b-421b-8d51-5bb50801cb55-kube-api-access-lctvl\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:23.331776 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.331729 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr"] Nov 03 20:36:23.336380 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.336351 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c60d2a-69c97f8cb7-txgnr"] Nov 03 20:36:23.756069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.756035 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b812c57-ec4a-44e0-bc19-f5c2498ed507" path="/var/lib/kubelet/pods/9b812c57-ec4a-44e0-bc19-f5c2498ed507/volumes" Nov 03 20:36:23.756449 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:23.756435 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc31109-670b-421b-8d51-5bb50801cb55" path="/var/lib/kubelet/pods/fdc31109-670b-421b-8d51-5bb50801cb55/volumes" Nov 03 20:36:26.274772 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.274719 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" containerID="cri-o://953b275330478d98d952bc48b9ea397bfb3a09a69ba174a8e6ac627786266ab5" gracePeriod=45 Nov 03 20:36:26.305039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.305008 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" containerID="cri-o://92871e60191a1ad43dbde6054dc33eebd1ac1fd2b4f5d720c04786783baa9c5e" gracePeriod=45 Nov 03 20:36:26.549783 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.549755 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:36:26.582540 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.582506 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:36:26.682777 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.682745 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a97a4814-275f-4b05-88a0-b2ab3cd6305c-kube-api-access-gz8bl\" (UniqueName: \"kubernetes.io/projected/a97a4814-275f-4b05-88a0-b2ab3cd6305c-kube-api-access-gz8bl\") pod \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " Nov 03 20:36:26.683175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.682820 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-model-cache\") pod \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " Nov 03 20:36:26.683175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.682959 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tls-certs\") pod \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " Nov 03 20:36:26.683175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683016 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tmp-dir\") pod \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " Nov 03 20:36:26.683175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683042 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tmp-dir\") pod \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " Nov 03 20:36:26.683175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683065 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-model-cache" pod "a97a4814-275f-4b05-88a0-b2ab3cd6305c" (UID: "a97a4814-275f-4b05-88a0-b2ab3cd6305c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.683175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683076 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-dshm\") pod \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " Nov 03 20:36:26.683175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683133 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-model-cache\") pod \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " Nov 03 20:36:26.683686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683192 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-home\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-home\") pod \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " Nov 03 20:36:26.683686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683233 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-home\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-home\") pod \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " Nov 03 20:36:26.683686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683307 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tmp-dir" pod "ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" (UID: "ea3405cc-6fa5-459c-a7b2-189a1b09c6d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.683686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683325 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-kube-api-access-7qfts\" (UniqueName: \"kubernetes.io/projected/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-kube-api-access-7qfts\") pod \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " Nov 03 20:36:26.683686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683351 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tls-certs\") pod \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\" (UID: \"a97a4814-275f-4b05-88a0-b2ab3cd6305c\") " Nov 03 20:36:26.683686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683329 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tmp-dir" pod "a97a4814-275f-4b05-88a0-b2ab3cd6305c" (UID: "a97a4814-275f-4b05-88a0-b2ab3cd6305c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.683686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683404 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-dshm\") pod \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\" (UID: \"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5\") " Nov 03 20:36:26.683686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683442 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-model-cache" pod "ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" (UID: "ea3405cc-6fa5-459c-a7b2-189a1b09c6d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.683686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683559 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-home" pod "ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" (UID: "ea3405cc-6fa5-459c-a7b2-189a1b09c6d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.683686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683594 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-home" pod "a97a4814-275f-4b05-88a0-b2ab3cd6305c" (UID: "a97a4814-275f-4b05-88a0-b2ab3cd6305c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.684271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683758 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.684271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683781 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.684271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683797 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.684271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683810 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.684271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683822 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.684271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.683839 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.685534 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.685505 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-dshm" pod "a97a4814-275f-4b05-88a0-b2ab3cd6305c" (UID: "a97a4814-275f-4b05-88a0-b2ab3cd6305c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.685655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.685629 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tls-certs" pod "a97a4814-275f-4b05-88a0-b2ab3cd6305c" (UID: "a97a4814-275f-4b05-88a0-b2ab3cd6305c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:26.685737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.685714 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tls-certs" pod "ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" (UID: "ea3405cc-6fa5-459c-a7b2-189a1b09c6d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:26.685782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.685742 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a97a4814-275f-4b05-88a0-b2ab3cd6305c-kube-api-access-gz8bl" pod "a97a4814-275f-4b05-88a0-b2ab3cd6305c" (UID: "a97a4814-275f-4b05-88a0-b2ab3cd6305c"). InnerVolumeSpecName "kube-api-access-gz8bl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:26.685855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.685836 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-kube-api-access-7qfts" pod "ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" (UID: "ea3405cc-6fa5-459c-a7b2-189a1b09c6d5"). InnerVolumeSpecName "kube-api-access-7qfts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:26.685927 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.685907 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-dshm" pod "ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" (UID: "ea3405cc-6fa5-459c-a7b2-189a1b09c6d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:26.784880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.784797 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qfts\" (UniqueName: \"kubernetes.io/projected/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-kube-api-access-7qfts\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.784880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.784829 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a97a4814-275f-4b05-88a0-b2ab3cd6305c-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.784880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.784840 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.784880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.784849 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz8bl\" (UniqueName: \"kubernetes.io/projected/a97a4814-275f-4b05-88a0-b2ab3cd6305c-kube-api-access-gz8bl\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.784880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.784872 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:26.784880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:26.784883 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a97a4814-275f-4b05-88a0-b2ab3cd6305c-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:27.036707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.036598 4457 generic.go:356] "Generic (PLEG): container finished" podID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerID="953b275330478d98d952bc48b9ea397bfb3a09a69ba174a8e6ac627786266ab5" exitCode=0 Nov 03 20:36:27.036707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.036652 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" Nov 03 20:36:27.036707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.036678 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" event={"ID":"a97a4814-275f-4b05-88a0-b2ab3cd6305c","Type":"ContainerDied","Data":"953b275330478d98d952bc48b9ea397bfb3a09a69ba174a8e6ac627786266ab5"} Nov 03 20:36:27.037074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.036727 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc" event={"ID":"a97a4814-275f-4b05-88a0-b2ab3cd6305c","Type":"ContainerDied","Data":"c6056f0291a4ac55028e128b4352d65264cf428ab80228f68f0cf7ab124233dd"} Nov 03 20:36:27.037074 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.036744 4457 scope.go:122] "RemoveContainer" containerID="953b275330478d98d952bc48b9ea397bfb3a09a69ba174a8e6ac627786266ab5" Nov 03 20:36:27.038349 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.038327 4457 generic.go:356] "Generic (PLEG): container finished" podID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerID="92871e60191a1ad43dbde6054dc33eebd1ac1fd2b4f5d720c04786783baa9c5e" exitCode=0 Nov 03 20:36:27.038489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.038405 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" event={"ID":"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5","Type":"ContainerDied","Data":"92871e60191a1ad43dbde6054dc33eebd1ac1fd2b4f5d720c04786783baa9c5e"} Nov 03 20:36:27.038489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.038418 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" Nov 03 20:36:27.038489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.038437 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm" event={"ID":"ea3405cc-6fa5-459c-a7b2-189a1b09c6d5","Type":"ContainerDied","Data":"de4fd98da4227dcb3c5d153f2b8778a18a8dec516253cce5157c4fab1a7237e2"} Nov 03 20:36:27.048192 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.048175 4457 scope.go:122] "RemoveContainer" containerID="953b275330478d98d952bc48b9ea397bfb3a09a69ba174a8e6ac627786266ab5" Nov 03 20:36:27.048478 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:36:27.048455 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"953b275330478d98d952bc48b9ea397bfb3a09a69ba174a8e6ac627786266ab5\": container with ID starting with 953b275330478d98d952bc48b9ea397bfb3a09a69ba174a8e6ac627786266ab5 not found: ID does not exist" containerID="953b275330478d98d952bc48b9ea397bfb3a09a69ba174a8e6ac627786266ab5" Nov 03 20:36:27.048575 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.048485 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"953b275330478d98d952bc48b9ea397bfb3a09a69ba174a8e6ac627786266ab5"} err="failed to get container status \"953b275330478d98d952bc48b9ea397bfb3a09a69ba174a8e6ac627786266ab5\": rpc error: code = NotFound desc = could not find container \"953b275330478d98d952bc48b9ea397bfb3a09a69ba174a8e6ac627786266ab5\": container with ID starting with 953b275330478d98d952bc48b9ea397bfb3a09a69ba174a8e6ac627786266ab5 not found: ID does not exist" Nov 03 20:36:27.048575 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.048501 4457 scope.go:122] "RemoveContainer" containerID="92871e60191a1ad43dbde6054dc33eebd1ac1fd2b4f5d720c04786783baa9c5e" Nov 03 20:36:27.057071 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.057051 4457 scope.go:122] "RemoveContainer" containerID="92871e60191a1ad43dbde6054dc33eebd1ac1fd2b4f5d720c04786783baa9c5e" Nov 03 20:36:27.057363 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:36:27.057316 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92871e60191a1ad43dbde6054dc33eebd1ac1fd2b4f5d720c04786783baa9c5e\": container with ID starting with 92871e60191a1ad43dbde6054dc33eebd1ac1fd2b4f5d720c04786783baa9c5e not found: ID does not exist" containerID="92871e60191a1ad43dbde6054dc33eebd1ac1fd2b4f5d720c04786783baa9c5e" Nov 03 20:36:27.057363 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.057345 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92871e60191a1ad43dbde6054dc33eebd1ac1fd2b4f5d720c04786783baa9c5e"} err="failed to get container status \"92871e60191a1ad43dbde6054dc33eebd1ac1fd2b4f5d720c04786783baa9c5e\": rpc error: code = NotFound desc = could not find container \"92871e60191a1ad43dbde6054dc33eebd1ac1fd2b4f5d720c04786783baa9c5e\": container with ID starting with 92871e60191a1ad43dbde6054dc33eebd1ac1fd2b4f5d720c04786783baa9c5e not found: ID does not exist" Nov 03 20:36:27.061060 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.061039 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc"] Nov 03 20:36:27.064875 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.064855 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/distinct-2-w0-b1b190-kserve-68cd4b5d47-cd5tc"] Nov 03 20:36:27.076788 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.076763 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm"] Nov 03 20:36:27.079264 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.079245 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-b1b190/simulator-w0-b1b190-kserve-7f9d69dd-4vcqm"] Nov 03 20:36:27.757283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.757243 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" path="/var/lib/kubelet/pods/a97a4814-275f-4b05-88a0-b2ab3cd6305c/volumes" Nov 03 20:36:27.757873 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:27.757854 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" path="/var/lib/kubelet/pods/ea3405cc-6fa5-459c-a7b2-189a1b09c6d5/volumes" Nov 03 20:36:31.962143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:31.962105 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" podUID="54c875b5-372c-42fe-828a-7605988b595d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 20:36:32.167583 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.167541 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt"] Nov 03 20:36:32.167867 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.167820 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" podUID="0149807d-7ed5-4e99-a457-04532e25b0a7" containerName="maas-api" containerID="cri-o://d37d76de3b09c5c7f0378319ea654cfd175593ce3227e0b7226910b42a804369" gracePeriod=30 Nov 03 20:36:32.432639 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.432616 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:36:32.540006 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.539923 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-metrics-certs\") pod \"0149807d-7ed5-4e99-a457-04532e25b0a7\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " Nov 03 20:36:32.540190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.540010 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-maas-api-tls\") pod \"0149807d-7ed5-4e99-a457-04532e25b0a7\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " Nov 03 20:36:32.540190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.540055 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0149807d-7ed5-4e99-a457-04532e25b0a7-kube-api-access-n5hh6\" (UniqueName: \"kubernetes.io/projected/0149807d-7ed5-4e99-a457-04532e25b0a7-kube-api-access-n5hh6\") pod \"0149807d-7ed5-4e99-a457-04532e25b0a7\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " Nov 03 20:36:32.540190 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.540115 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0149807d-7ed5-4e99-a457-04532e25b0a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149807d-7ed5-4e99-a457-04532e25b0a7-openshift-service-ca\") pod \"0149807d-7ed5-4e99-a457-04532e25b0a7\" (UID: \"0149807d-7ed5-4e99-a457-04532e25b0a7\") " Nov 03 20:36:32.540699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.540672 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0149807d-7ed5-4e99-a457-04532e25b0a7-openshift-service-ca" pod "0149807d-7ed5-4e99-a457-04532e25b0a7" (UID: "0149807d-7ed5-4e99-a457-04532e25b0a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:32.542932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.542907 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-maas-api-tls" pod "0149807d-7ed5-4e99-a457-04532e25b0a7" (UID: "0149807d-7ed5-4e99-a457-04532e25b0a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:32.543064 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.542990 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0149807d-7ed5-4e99-a457-04532e25b0a7-kube-api-access-n5hh6" pod "0149807d-7ed5-4e99-a457-04532e25b0a7" (UID: "0149807d-7ed5-4e99-a457-04532e25b0a7"). InnerVolumeSpecName "kube-api-access-n5hh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:32.544623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.544599 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-metrics-certs" pod "0149807d-7ed5-4e99-a457-04532e25b0a7" (UID: "0149807d-7ed5-4e99-a457-04532e25b0a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:32.641984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.641943 4457 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0149807d-7ed5-4e99-a457-04532e25b0a7-openshift-service-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:32.641984 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.641987 4457 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-metrics-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:32.642245 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.642002 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0149807d-7ed5-4e99-a457-04532e25b0a7-maas-api-tls\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:32.642245 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:32.642016 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5hh6\" (UniqueName: \"kubernetes.io/projected/0149807d-7ed5-4e99-a457-04532e25b0a7-kube-api-access-n5hh6\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:33.100910 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:33.100870 4457 generic.go:356] "Generic (PLEG): container finished" podID="0149807d-7ed5-4e99-a457-04532e25b0a7" containerID="d37d76de3b09c5c7f0378319ea654cfd175593ce3227e0b7226910b42a804369" exitCode=0 Nov 03 20:36:33.101487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:33.100932 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" Nov 03 20:36:33.101487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:33.100971 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" event={"ID":"0149807d-7ed5-4e99-a457-04532e25b0a7","Type":"ContainerDied","Data":"d37d76de3b09c5c7f0378319ea654cfd175593ce3227e0b7226910b42a804369"} Nov 03 20:36:33.101487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:33.101031 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt" event={"ID":"0149807d-7ed5-4e99-a457-04532e25b0a7","Type":"ContainerDied","Data":"c295b80d6ca3b7ed545090a18c38e91d4b7f0dd86b3a17c3ed91b68b4de3e511"} Nov 03 20:36:33.101487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:33.101062 4457 scope.go:122] "RemoveContainer" containerID="d37d76de3b09c5c7f0378319ea654cfd175593ce3227e0b7226910b42a804369" Nov 03 20:36:33.118858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:33.118830 4457 scope.go:122] "RemoveContainer" containerID="d37d76de3b09c5c7f0378319ea654cfd175593ce3227e0b7226910b42a804369" Nov 03 20:36:33.119495 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:36:33.119427 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37d76de3b09c5c7f0378319ea654cfd175593ce3227e0b7226910b42a804369\": container with ID starting with d37d76de3b09c5c7f0378319ea654cfd175593ce3227e0b7226910b42a804369 not found: ID does not exist" containerID="d37d76de3b09c5c7f0378319ea654cfd175593ce3227e0b7226910b42a804369" Nov 03 20:36:33.119495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:33.119467 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37d76de3b09c5c7f0378319ea654cfd175593ce3227e0b7226910b42a804369"} err="failed to get container status \"d37d76de3b09c5c7f0378319ea654cfd175593ce3227e0b7226910b42a804369\": rpc error: code = NotFound desc = could not find container \"d37d76de3b09c5c7f0378319ea654cfd175593ce3227e0b7226910b42a804369\": container with ID starting with d37d76de3b09c5c7f0378319ea654cfd175593ce3227e0b7226910b42a804369 not found: ID does not exist" Nov 03 20:36:33.131266 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:33.131241 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt"] Nov 03 20:36:33.137900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:33.137876 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b1b190-b47cd6fdd-6spnt"] Nov 03 20:36:33.757121 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:33.757085 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0149807d-7ed5-4e99-a457-04532e25b0a7" path="/var/lib/kubelet/pods/0149807d-7ed5-4e99-a457-04532e25b0a7/volumes" Nov 03 20:36:34.022951 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:34.022863 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc"] Nov 03 20:36:34.026856 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:34.026828 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f757389a-24stc"] Nov 03 20:36:35.756321 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:35.756283 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd78a7f-4170-42c1-9bb6-919b7327325d" path="/var/lib/kubelet/pods/3bd78a7f-4170-42c1-9bb6-919b7327325d/volumes" Nov 03 20:36:36.145757 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:36:36.145718 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:36:39.148932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:39.148888 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f"] Nov 03 20:36:39.149474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:39.149174 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" containerID="cri-o://4c334a09f7b1cde8c49a9c999772d1cb646484e8ea8cbea90388ffa69e2f01a7" gracePeriod=30 Nov 03 20:36:41.850610 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:41.850572 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-69l6m"] Nov 03 20:36:41.851091 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:41.850906 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" podUID="2bb0a4bf-7a65-40ea-9a42-6d1d89c76645" containerName="manager" containerID="cri-o://24ba90871525bf113d450bcb9f142c5c0c55f045bca1fe3ea872a4138860370c" gracePeriod=30 Nov 03 20:36:41.966902 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:41.966880 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" Nov 03 20:36:42.008335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.008003 4457 patch_prober.go:29] interesting pod/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:36:42.008335 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.008066 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:36:42.034974 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.034942 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c875b5-372c-42fe-828a-7605988b595d-kube-api-access-zqd87\" (UniqueName: \"kubernetes.io/projected/54c875b5-372c-42fe-828a-7605988b595d-kube-api-access-zqd87\") pod \"54c875b5-372c-42fe-828a-7605988b595d\" (UID: \"54c875b5-372c-42fe-828a-7605988b595d\") " Nov 03 20:36:42.037714 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.037679 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c875b5-372c-42fe-828a-7605988b595d-kube-api-access-zqd87" pod "54c875b5-372c-42fe-828a-7605988b595d" (UID: "54c875b5-372c-42fe-828a-7605988b595d"). InnerVolumeSpecName "kube-api-access-zqd87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:42.127878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.127848 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:36:42.135887 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.135828 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-webhook-cert\") pod \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\" (UID: \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\") " Nov 03 20:36:42.136036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.135900 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c875b5-372c-42fe-828a-7605988b595d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c875b5-372c-42fe-828a-7605988b595d-plugins-config-volume\") pod \"54c875b5-372c-42fe-828a-7605988b595d\" (UID: \"54c875b5-372c-42fe-828a-7605988b595d\") " Nov 03 20:36:42.136036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.135981 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-metrics-certs\") pod \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\" (UID: \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\") " Nov 03 20:36:42.136036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.136020 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-kube-api-access-qf7rc\" (UniqueName: \"kubernetes.io/projected/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-kube-api-access-qf7rc\") pod \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\" (UID: \"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645\") " Nov 03 20:36:42.136265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.136240 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c875b5-372c-42fe-828a-7605988b595d-plugins-config-volume" pod "54c875b5-372c-42fe-828a-7605988b595d" (UID: "54c875b5-372c-42fe-828a-7605988b595d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:42.136452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.136434 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54c875b5-372c-42fe-828a-7605988b595d-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:42.136521 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.136456 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqd87\" (UniqueName: \"kubernetes.io/projected/54c875b5-372c-42fe-828a-7605988b595d-kube-api-access-zqd87\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:42.137903 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.137882 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-webhook-cert" pod "2bb0a4bf-7a65-40ea-9a42-6d1d89c76645" (UID: "2bb0a4bf-7a65-40ea-9a42-6d1d89c76645"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:42.137982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.137938 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-metrics-certs" pod "2bb0a4bf-7a65-40ea-9a42-6d1d89c76645" (UID: "2bb0a4bf-7a65-40ea-9a42-6d1d89c76645"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:36:42.137982 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.137948 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-kube-api-access-qf7rc" pod "2bb0a4bf-7a65-40ea-9a42-6d1d89c76645" (UID: "2bb0a4bf-7a65-40ea-9a42-6d1d89c76645"). InnerVolumeSpecName "kube-api-access-qf7rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:42.184883 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.184843 4457 generic.go:356] "Generic (PLEG): container finished" podID="2bb0a4bf-7a65-40ea-9a42-6d1d89c76645" containerID="24ba90871525bf113d450bcb9f142c5c0c55f045bca1fe3ea872a4138860370c" exitCode=0 Nov 03 20:36:42.185091 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.184898 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" Nov 03 20:36:42.185091 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.184931 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" event={"ID":"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645","Type":"ContainerDied","Data":"24ba90871525bf113d450bcb9f142c5c0c55f045bca1fe3ea872a4138860370c"} Nov 03 20:36:42.185091 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.185005 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6dd9f77dd9-69l6m" event={"ID":"2bb0a4bf-7a65-40ea-9a42-6d1d89c76645","Type":"ContainerDied","Data":"f31fd8fbd1ceeecb37218ec94d1c4007b90c07fc020c6b38f3d22d72541644d6"} Nov 03 20:36:42.185091 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.185037 4457 scope.go:122] "RemoveContainer" containerID="24ba90871525bf113d450bcb9f142c5c0c55f045bca1fe3ea872a4138860370c" Nov 03 20:36:42.186782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.186752 4457 generic.go:356] "Generic (PLEG): container finished" podID="54c875b5-372c-42fe-828a-7605988b595d" containerID="fe80e7cabbbff7b9e436b7e98a7b620a4b1da0df852dfb8836c8fb7c09258cfb" exitCode=0 Nov 03 20:36:42.186949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.186805 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" Nov 03 20:36:42.186949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.186839 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" event={"ID":"54c875b5-372c-42fe-828a-7605988b595d","Type":"ContainerDied","Data":"fe80e7cabbbff7b9e436b7e98a7b620a4b1da0df852dfb8836c8fb7c09258cfb"} Nov 03 20:36:42.186949 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.186891 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" event={"ID":"54c875b5-372c-42fe-828a-7605988b595d","Type":"ContainerDied","Data":"24089de8658e281f5755ffdfe713fc1e17b519d7629f385419e3d7f73639ec17"} Nov 03 20:36:42.201342 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.201313 4457 scope.go:122] "RemoveContainer" containerID="4cacf177178e5818c08b46ac32f56a2bf1c417ae930a3fa3388b54abd1e4dc95" Nov 03 20:36:42.212499 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.212471 4457 scope.go:122] "RemoveContainer" containerID="24ba90871525bf113d450bcb9f142c5c0c55f045bca1fe3ea872a4138860370c" Nov 03 20:36:42.212780 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:36:42.212762 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ba90871525bf113d450bcb9f142c5c0c55f045bca1fe3ea872a4138860370c\": container with ID starting with 24ba90871525bf113d450bcb9f142c5c0c55f045bca1fe3ea872a4138860370c not found: ID does not exist" containerID="24ba90871525bf113d450bcb9f142c5c0c55f045bca1fe3ea872a4138860370c" Nov 03 20:36:42.212843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.212789 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ba90871525bf113d450bcb9f142c5c0c55f045bca1fe3ea872a4138860370c"} err="failed to get container status \"24ba90871525bf113d450bcb9f142c5c0c55f045bca1fe3ea872a4138860370c\": rpc error: code = NotFound desc = could not find container \"24ba90871525bf113d450bcb9f142c5c0c55f045bca1fe3ea872a4138860370c\": container with ID starting with 24ba90871525bf113d450bcb9f142c5c0c55f045bca1fe3ea872a4138860370c not found: ID does not exist" Nov 03 20:36:42.212843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.212807 4457 scope.go:122] "RemoveContainer" containerID="4cacf177178e5818c08b46ac32f56a2bf1c417ae930a3fa3388b54abd1e4dc95" Nov 03 20:36:42.213034 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:36:42.213018 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cacf177178e5818c08b46ac32f56a2bf1c417ae930a3fa3388b54abd1e4dc95\": container with ID starting with 4cacf177178e5818c08b46ac32f56a2bf1c417ae930a3fa3388b54abd1e4dc95 not found: ID does not exist" containerID="4cacf177178e5818c08b46ac32f56a2bf1c417ae930a3fa3388b54abd1e4dc95" Nov 03 20:36:42.213078 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.213041 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cacf177178e5818c08b46ac32f56a2bf1c417ae930a3fa3388b54abd1e4dc95"} err="failed to get container status \"4cacf177178e5818c08b46ac32f56a2bf1c417ae930a3fa3388b54abd1e4dc95\": rpc error: code = NotFound desc = could not find container \"4cacf177178e5818c08b46ac32f56a2bf1c417ae930a3fa3388b54abd1e4dc95\": container with ID starting with 4cacf177178e5818c08b46ac32f56a2bf1c417ae930a3fa3388b54abd1e4dc95 not found: ID does not exist" Nov 03 20:36:42.213078 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.213054 4457 scope.go:122] "RemoveContainer" containerID="fe80e7cabbbff7b9e436b7e98a7b620a4b1da0df852dfb8836c8fb7c09258cfb" Nov 03 20:36:42.217666 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.217641 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-69l6m"] Nov 03 20:36:42.224486 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.224468 4457 scope.go:122] "RemoveContainer" containerID="fe80e7cabbbff7b9e436b7e98a7b620a4b1da0df852dfb8836c8fb7c09258cfb" Nov 03 20:36:42.224815 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:36:42.224794 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe80e7cabbbff7b9e436b7e98a7b620a4b1da0df852dfb8836c8fb7c09258cfb\": container with ID starting with fe80e7cabbbff7b9e436b7e98a7b620a4b1da0df852dfb8836c8fb7c09258cfb not found: ID does not exist" containerID="fe80e7cabbbff7b9e436b7e98a7b620a4b1da0df852dfb8836c8fb7c09258cfb" Nov 03 20:36:42.224878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.224822 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe80e7cabbbff7b9e436b7e98a7b620a4b1da0df852dfb8836c8fb7c09258cfb"} err="failed to get container status \"fe80e7cabbbff7b9e436b7e98a7b620a4b1da0df852dfb8836c8fb7c09258cfb\": rpc error: code = NotFound desc = could not find container \"fe80e7cabbbff7b9e436b7e98a7b620a4b1da0df852dfb8836c8fb7c09258cfb\": container with ID starting with fe80e7cabbbff7b9e436b7e98a7b620a4b1da0df852dfb8836c8fb7c09258cfb not found: ID does not exist" Nov 03 20:36:42.225076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.225058 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6dd9f77dd9-69l6m"] Nov 03 20:36:42.235205 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.235172 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4"] Nov 03 20:36:42.237324 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.237292 4457 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-metrics-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:42.237443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.237324 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf7rc\" (UniqueName: \"kubernetes.io/projected/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-kube-api-access-qf7rc\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:42.237443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.237344 4457 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645-webhook-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:42.241161 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.241134 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4"] Nov 03 20:36:42.961738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:42.961636 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b1b190-7964c4895-qgbd4" podUID="54c875b5-372c-42fe-828a-7605988b595d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.49:50052\" within 1s: context deadline exceeded" Nov 03 20:36:42.961738 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:36:42.961704 4457 logging.go:55] [core] [Channel #7210 SubChannel #7211]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.49:50052", ServerName: "10.133.0.49:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.49:50052: operation was canceled" Nov 03 20:36:43.759983 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:43.759946 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bb0a4bf-7a65-40ea-9a42-6d1d89c76645" path="/var/lib/kubelet/pods/2bb0a4bf-7a65-40ea-9a42-6d1d89c76645/volumes" Nov 03 20:36:43.760538 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:43.760517 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c875b5-372c-42fe-828a-7605988b595d" path="/var/lib/kubelet/pods/54c875b5-372c-42fe-828a-7605988b595d/volumes" Nov 03 20:36:44.555085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.555061 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:36:44.612366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.606750 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-istio-token\" (UniqueName: \"kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-istio-token\") pod \"942423a8-2c01-4ef8-82df-cf8892de3585\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " Nov 03 20:36:44.612366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.606861 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-credential-socket\") pod \"942423a8-2c01-4ef8-82df-cf8892de3585\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " Nov 03 20:36:44.612366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.606925 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/942423a8-2c01-4ef8-82df-cf8892de3585-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942423a8-2c01-4ef8-82df-cf8892de3585-istio-podinfo\") pod \"942423a8-2c01-4ef8-82df-cf8892de3585\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " Nov 03 20:36:44.612366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.606960 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942423a8-2c01-4ef8-82df-cf8892de3585-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942423a8-2c01-4ef8-82df-cf8892de3585-istiod-ca-cert\") pod \"942423a8-2c01-4ef8-82df-cf8892de3585\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " Nov 03 20:36:44.612366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.606993 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-certs\") pod \"942423a8-2c01-4ef8-82df-cf8892de3585\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " Nov 03 20:36:44.612366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.607036 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-socket\") pod \"942423a8-2c01-4ef8-82df-cf8892de3585\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " Nov 03 20:36:44.612366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.607091 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-kube-api-access-lcx9n\" (UniqueName: \"kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-kube-api-access-lcx9n\") pod \"942423a8-2c01-4ef8-82df-cf8892de3585\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " Nov 03 20:36:44.612366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.607122 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-data\") pod \"942423a8-2c01-4ef8-82df-cf8892de3585\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " Nov 03 20:36:44.612366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.607188 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-envoy\") pod \"942423a8-2c01-4ef8-82df-cf8892de3585\" (UID: \"942423a8-2c01-4ef8-82df-cf8892de3585\") " Nov 03 20:36:44.612366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.608349 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942423a8-2c01-4ef8-82df-cf8892de3585-istiod-ca-cert" pod "942423a8-2c01-4ef8-82df-cf8892de3585" (UID: "942423a8-2c01-4ef8-82df-cf8892de3585"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:36:44.612366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.609370 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-credential-socket" pod "942423a8-2c01-4ef8-82df-cf8892de3585" (UID: "942423a8-2c01-4ef8-82df-cf8892de3585"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.612366 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.611065 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-certs" pod "942423a8-2c01-4ef8-82df-cf8892de3585" (UID: "942423a8-2c01-4ef8-82df-cf8892de3585"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.613169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.612640 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-istio-token" pod "942423a8-2c01-4ef8-82df-cf8892de3585" (UID: "942423a8-2c01-4ef8-82df-cf8892de3585"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:44.613169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.612768 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-data" pod "942423a8-2c01-4ef8-82df-cf8892de3585" (UID: "942423a8-2c01-4ef8-82df-cf8892de3585"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.626548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.626513 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-socket" pod "942423a8-2c01-4ef8-82df-cf8892de3585" (UID: "942423a8-2c01-4ef8-82df-cf8892de3585"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.628443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.628275 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/942423a8-2c01-4ef8-82df-cf8892de3585-istio-podinfo" pod "942423a8-2c01-4ef8-82df-cf8892de3585" (UID: "942423a8-2c01-4ef8-82df-cf8892de3585"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:36:44.628940 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.628911 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-kube-api-access-lcx9n" pod "942423a8-2c01-4ef8-82df-cf8892de3585" (UID: "942423a8-2c01-4ef8-82df-cf8892de3585"). InnerVolumeSpecName "kube-api-access-lcx9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:36:44.629077 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.629056 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-envoy" pod "942423a8-2c01-4ef8-82df-cf8892de3585" (UID: "942423a8-2c01-4ef8-82df-cf8892de3585"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:36:44.718578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.710849 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.718578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.710882 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.718578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.710893 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.718578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.710902 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942423a8-2c01-4ef8-82df-cf8892de3585-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.718578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.710910 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942423a8-2c01-4ef8-82df-cf8892de3585-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.718578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.710920 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.718578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.710929 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.718578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.710937 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcx9n\" (UniqueName: \"kubernetes.io/projected/942423a8-2c01-4ef8-82df-cf8892de3585-kube-api-access-lcx9n\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:44.718578 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:44.710951 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942423a8-2c01-4ef8-82df-cf8892de3585-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:36:45.229334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:45.229298 4457 generic.go:356] "Generic (PLEG): container finished" podID="942423a8-2c01-4ef8-82df-cf8892de3585" containerID="4c334a09f7b1cde8c49a9c999772d1cb646484e8ea8cbea90388ffa69e2f01a7" exitCode=0 Nov 03 20:36:45.229574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:45.229358 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" Nov 03 20:36:45.229574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:45.229405 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" event={"ID":"942423a8-2c01-4ef8-82df-cf8892de3585","Type":"ContainerDied","Data":"4c334a09f7b1cde8c49a9c999772d1cb646484e8ea8cbea90388ffa69e2f01a7"} Nov 03 20:36:45.229574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:45.229479 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f" event={"ID":"942423a8-2c01-4ef8-82df-cf8892de3585","Type":"ContainerDied","Data":"15ad228c1c72705939d5fbde867b76d995b3f0a927969e9bfc06505fffdf0331"} Nov 03 20:36:45.229574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:45.229510 4457 scope.go:122] "RemoveContainer" containerID="4c334a09f7b1cde8c49a9c999772d1cb646484e8ea8cbea90388ffa69e2f01a7" Nov 03 20:36:45.242181 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:45.242152 4457 scope.go:122] "RemoveContainer" containerID="bd1119a197ada5445a2ce8cdb33836c21d6ee8ab27178de9b21ead66762627cb" Nov 03 20:36:45.261686 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:45.261549 4457 scope.go:122] "RemoveContainer" containerID="4c334a09f7b1cde8c49a9c999772d1cb646484e8ea8cbea90388ffa69e2f01a7" Nov 03 20:36:45.262302 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:36:45.262155 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c334a09f7b1cde8c49a9c999772d1cb646484e8ea8cbea90388ffa69e2f01a7\": container with ID starting with 4c334a09f7b1cde8c49a9c999772d1cb646484e8ea8cbea90388ffa69e2f01a7 not found: ID does not exist" containerID="4c334a09f7b1cde8c49a9c999772d1cb646484e8ea8cbea90388ffa69e2f01a7" Nov 03 20:36:45.262302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:45.262196 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c334a09f7b1cde8c49a9c999772d1cb646484e8ea8cbea90388ffa69e2f01a7"} err="failed to get container status \"4c334a09f7b1cde8c49a9c999772d1cb646484e8ea8cbea90388ffa69e2f01a7\": rpc error: code = NotFound desc = could not find container \"4c334a09f7b1cde8c49a9c999772d1cb646484e8ea8cbea90388ffa69e2f01a7\": container with ID starting with 4c334a09f7b1cde8c49a9c999772d1cb646484e8ea8cbea90388ffa69e2f01a7 not found: ID does not exist" Nov 03 20:36:45.262302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:45.262221 4457 scope.go:122] "RemoveContainer" containerID="bd1119a197ada5445a2ce8cdb33836c21d6ee8ab27178de9b21ead66762627cb" Nov 03 20:36:45.262906 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:36:45.262846 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1119a197ada5445a2ce8cdb33836c21d6ee8ab27178de9b21ead66762627cb\": container with ID starting with bd1119a197ada5445a2ce8cdb33836c21d6ee8ab27178de9b21ead66762627cb not found: ID does not exist" containerID="bd1119a197ada5445a2ce8cdb33836c21d6ee8ab27178de9b21ead66762627cb" Nov 03 20:36:45.262906 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:45.262880 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1119a197ada5445a2ce8cdb33836c21d6ee8ab27178de9b21ead66762627cb"} err="failed to get container status \"bd1119a197ada5445a2ce8cdb33836c21d6ee8ab27178de9b21ead66762627cb\": rpc error: code = NotFound desc = could not find container \"bd1119a197ada5445a2ce8cdb33836c21d6ee8ab27178de9b21ead66762627cb\": container with ID starting with bd1119a197ada5445a2ce8cdb33836c21d6ee8ab27178de9b21ead66762627cb not found: ID does not exist" Nov 03 20:36:45.278886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:45.278858 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f"] Nov 03 20:36:45.284655 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:45.284616 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-b1b190-openshift-default-6647d74f7f-tq66f"] Nov 03 20:36:45.755398 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:36:45.755352 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" path="/var/lib/kubelet/pods/942423a8-2c01-4ef8-82df-cf8892de3585/volumes" Nov 03 20:37:02.874999 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:02.874963 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:37:02.875589 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:02.875569 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:37:02.890256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:02.890225 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:37:02.890461 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:02.890424 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:37:06.145009 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:37:06.144971 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:37:08.186448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:08.186405 4457 scope.go:122] "RemoveContainer" containerID="f499fcbd744a8309a84a3cf1f7cbb887981bce933155ed72a98928d40c5a011f" Nov 03 20:37:08.197471 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:08.197446 4457 scope.go:122] "RemoveContainer" containerID="d4677b943b46a919d67bc54ac8bd765b502448fedc2c5f66d471f23938455081" Nov 03 20:37:13.093858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.093818 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd"] Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094178 4457 cpu_manager.go:414] "containerMap: removing container" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094189 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094204 4457 cpu_manager.go:414] "containerMap: removing container" podUID="4de4b055-a669-4760-ba03-041291c506fd" containerName="payload-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094209 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de4b055-a669-4760-ba03-041291c506fd" containerName="payload-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094218 4457 cpu_manager.go:414] "containerMap: removing container" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094224 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094234 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d0ca0910-85fc-4163-a801-3f58ea5578e1" containerName="revoke-keys" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094242 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0ca0910-85fc-4163-a801-3f58ea5578e1" containerName="revoke-keys" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094251 4457 cpu_manager.go:414] "containerMap: removing container" podUID="7fa9c354-722f-46a0-be30-c4683f7e2254" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094259 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa9c354-722f-46a0-be30-c4683f7e2254" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094273 4457 cpu_manager.go:414] "containerMap: removing container" podUID="6cf12593-ef2a-4848-be72-ce57d759c2e9" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094278 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf12593-ef2a-4848-be72-ce57d759c2e9" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094295 4457 cpu_manager.go:414] "containerMap: removing container" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094302 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094311 4457 cpu_manager.go:414] "containerMap: removing container" podUID="896c36e5-f4a0-479e-b3c9-7f16881b272e" containerName="payload-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094318 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896c36e5-f4a0-479e-b3c9-7f16881b272e" containerName="payload-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094333 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c0e677dc-c502-43a4-9bdf-d7983a6f9b1b" containerName="payload-pre-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094341 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e677dc-c502-43a4-9bdf-d7983a6f9b1b" containerName="payload-pre-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094352 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c67ccec4-726a-4e35-a85f-baf6ff097782" containerName="maas-api" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094361 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67ccec4-726a-4e35-a85f-baf6ff097782" containerName="maas-api" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094372 4457 cpu_manager.go:414] "containerMap: removing container" podUID="92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094380 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094405 4457 cpu_manager.go:414] "containerMap: removing container" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094410 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094421 4457 cpu_manager.go:414] "containerMap: removing container" podUID="1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f" containerName="payload-pre-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094448 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f" containerName="payload-pre-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094459 4457 cpu_manager.go:414] "containerMap: removing container" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094468 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094479 4457 cpu_manager.go:414] "containerMap: removing container" podUID="54c875b5-372c-42fe-828a-7605988b595d" containerName="payload-pre-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094487 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c875b5-372c-42fe-828a-7605988b595d" containerName="payload-pre-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094497 4457 cpu_manager.go:414] "containerMap: removing container" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094506 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094515 4457 cpu_manager.go:414] "containerMap: removing container" podUID="fdc31109-670b-421b-8d51-5bb50801cb55" containerName="payload-pre-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094520 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc31109-670b-421b-8d51-5bb50801cb55" containerName="payload-pre-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094527 4457 cpu_manager.go:414] "containerMap: removing container" podUID="250b873d-632c-4169-b357-77f6d8b9e9b6" containerName="payload-pre-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094532 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250b873d-632c-4169-b357-77f6d8b9e9b6" containerName="payload-pre-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094540 4457 cpu_manager.go:414] "containerMap: removing container" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094548 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094560 4457 cpu_manager.go:414] "containerMap: removing container" podUID="97f3184c-7322-4684-94e1-9e5e4eecd6e7" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094568 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f3184c-7322-4684-94e1-9e5e4eecd6e7" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094578 4457 cpu_manager.go:414] "containerMap: removing container" podUID="32bb3d1e-c3a2-401a-be41-dc09cebde931" containerName="payload-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094586 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32bb3d1e-c3a2-401a-be41-dc09cebde931" containerName="payload-processing" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094595 4457 cpu_manager.go:414] "containerMap: removing container" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094600 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" Nov 03 20:37:13.094576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094610 4457 cpu_manager.go:414] "containerMap: removing container" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094617 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094628 4457 cpu_manager.go:414] "containerMap: removing container" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094634 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094644 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094649 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094657 4457 cpu_manager.go:414] "containerMap: removing container" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094663 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094674 4457 cpu_manager.go:414] "containerMap: removing container" podUID="37cbe749-2e98-4766-b527-8aa40a28e2b2" containerName="payload-pre-processing" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094681 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cbe749-2e98-4766-b527-8aa40a28e2b2" containerName="payload-pre-processing" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094696 4457 cpu_manager.go:414] "containerMap: removing container" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094702 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094711 4457 cpu_manager.go:414] "containerMap: removing container" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094951 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094974 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2bb0a4bf-7a65-40ea-9a42-6d1d89c76645" containerName="manager" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094979 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb0a4bf-7a65-40ea-9a42-6d1d89c76645" containerName="manager" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094990 4457 cpu_manager.go:414] "containerMap: removing container" podUID="2bb0a4bf-7a65-40ea-9a42-6d1d89c76645" containerName="manager" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.094995 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bb0a4bf-7a65-40ea-9a42-6d1d89c76645" containerName="manager" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095001 4457 cpu_manager.go:414] "containerMap: removing container" podUID="0149807d-7ed5-4e99-a457-04532e25b0a7" containerName="maas-api" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095005 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0149807d-7ed5-4e99-a457-04532e25b0a7" containerName="maas-api" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095018 4457 cpu_manager.go:414] "containerMap: removing container" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095025 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095119 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e677dc-c502-43a4-9bdf-d7983a6f9b1b" containerName="payload-pre-processing" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095127 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67ccec4-726a-4e35-a85f-baf6ff097782" containerName="maas-api" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095133 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="92dfdb31-f6cb-4e35-a1c4-4dffb94f37ab" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095140 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="008491db-6f31-4ea1-9cbf-f9f779af78ea" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095148 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbab914-a9eb-40dd-b9ba-74ed84ea8d0f" containerName="payload-pre-processing" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095154 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="c209ddd2-c195-4bd5-ba52-b6f13b6a35a1" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095160 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c875b5-372c-42fe-828a-7605988b595d" containerName="payload-pre-processing" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095165 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a5de13-728b-4a75-8990-807d1d3811e4" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095171 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc31109-670b-421b-8d51-5bb50801cb55" containerName="payload-pre-processing" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095176 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="250b873d-632c-4169-b357-77f6d8b9e9b6" containerName="payload-pre-processing" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095183 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095189 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f3184c-7322-4684-94e1-9e5e4eecd6e7" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095196 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="32bb3d1e-c3a2-401a-be41-dc09cebde931" containerName="payload-processing" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095204 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14a467c-de34-4c09-837d-ef5fb99986b7" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095212 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f4267e-0c59-4326-972b-b0925e478929" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095220 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095228 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3405cc-6fa5-459c-a7b2-189a1b09c6d5" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095236 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095243 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cbe749-2e98-4766-b527-8aa40a28e2b2" containerName="payload-pre-processing" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095252 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="942423a8-2c01-4ef8-82df-cf8892de3585" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095260 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6b8b440-7a7c-4713-bce7-529cf543c29d" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095268 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb0a4bf-7a65-40ea-9a42-6d1d89c76645" containerName="manager" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095277 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bb0a4bf-7a65-40ea-9a42-6d1d89c76645" containerName="manager" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095284 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="0149807d-7ed5-4e99-a457-04532e25b0a7" containerName="maas-api" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095291 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="a97a4814-275f-4b05-88a0-b2ab3cd6305c" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095301 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de4b055-a669-4760-ba03-041291c506fd" containerName="payload-processing" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095308 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b0efa8-3782-4cab-be92-a05fa383e808" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095316 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0ca0910-85fc-4163-a801-3f58ea5578e1" containerName="revoke-keys" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095323 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa9c354-722f-46a0-be30-c4683f7e2254" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095330 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf12593-ef2a-4848-be72-ce57d759c2e9" containerName="istio-proxy" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095340 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="22be1486-517e-4055-8e00-c24c944eb960" containerName="main" Nov 03 20:37:13.096293 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.095347 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="896c36e5-f4a0-479e-b3c9-7f16881b272e" containerName="payload-processing" Nov 03 20:37:13.100953 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.100931 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.110549 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.110519 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd"] Nov 03 20:37:13.283487 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.283444 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.283710 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.283513 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.283710 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.283598 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qh4\" (UniqueName: \"kubernetes.io/projected/2e3b005c-f816-4fbd-8812-b007b7fce9e7-kube-api-access-p5qh4\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.283710 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.283669 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.283879 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.283711 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.283879 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.283756 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.283879 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.283835 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.283879 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.283872 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.284045 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.283895 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.384542 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.384458 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.384542 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.384500 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.384542 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.384526 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.384839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.384549 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.384839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.384589 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.384839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.384609 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qh4\" (UniqueName: \"kubernetes.io/projected/2e3b005c-f816-4fbd-8812-b007b7fce9e7-kube-api-access-p5qh4\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.384839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.384630 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.384839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.384659 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.384839 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.384701 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.385219 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.384997 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.385219 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.385107 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.385219 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.385177 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.385398 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.385216 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.385475 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.385419 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.386882 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.386856 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.387096 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.387079 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.396874 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.396846 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2e3b005c-f816-4fbd-8812-b007b7fce9e7-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.397046 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.397029 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qh4\" (UniqueName: \"kubernetes.io/projected/2e3b005c-f816-4fbd-8812-b007b7fce9e7-kube-api-access-p5qh4\") pod \"maas-default-gateway-openshift-default-77f556d647-n42qd\" (UID: \"2e3b005c-f816-4fbd-8812-b007b7fce9e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.417749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.417728 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:13.702642 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.702616 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd"] Nov 03 20:37:13.704618 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:37:13.704586 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3b005c_f816_4fbd_8812_b007b7fce9e7.slice/crio-6cf9ae520bfda7e14057dc95f7c08e621d95aba1b75324a5602cde9770768dca WatchSource:0}: Error finding container 6cf9ae520bfda7e14057dc95f7c08e621d95aba1b75324a5602cde9770768dca: Status 404 returned error can't find the container with id 6cf9ae520bfda7e14057dc95f7c08e621d95aba1b75324a5602cde9770768dca Nov 03 20:37:13.706917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.706885 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:37:13.707007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.706955 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:37:13.707007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:13.706985 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:37:14.040827 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:14.040727 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2"] Nov 03 20:37:14.043749 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:14.043713 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1220971-4h5d2"] Nov 03 20:37:14.441962 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:14.441923 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" event={"ID":"2e3b005c-f816-4fbd-8812-b007b7fce9e7","Type":"ContainerStarted","Data":"4a6ab709aaccab685c126c441b62ffca04991ee4a43c6a6d1209265e0cc7bdfd"} Nov 03 20:37:14.442449 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:14.441983 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" event={"ID":"2e3b005c-f816-4fbd-8812-b007b7fce9e7","Type":"ContainerStarted","Data":"6cf9ae520bfda7e14057dc95f7c08e621d95aba1b75324a5602cde9770768dca"} Nov 03 20:37:14.477410 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:14.474232 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" podStartSLOduration=1.474212739 podStartE2EDuration="1.474212739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:37:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:37:14.470922971 +0000 UTC m=+2412.846519721" watchObservedRunningTime="2026-11-03 20:37:14.474212739 +0000 UTC m=+2412.849809489" Nov 03 20:37:15.023315 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:15.023276 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72"] Nov 03 20:37:15.025683 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:15.025659 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383998a9-gbc72"] Nov 03 20:37:15.417981 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:15.417939 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:15.419339 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:15.419312 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-n42qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 03 20:37:15.419511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:15.419360 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" podUID="2e3b005c-f816-4fbd-8812-b007b7fce9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 03 20:37:15.761458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:15.755690 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4679afb7-c483-44e4-9ae9-c6da4dffbd44" path="/var/lib/kubelet/pods/4679afb7-c483-44e4-9ae9-c6da4dffbd44/volumes" Nov 03 20:37:15.761458 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:15.756333 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1" path="/var/lib/kubelet/pods/d1a21410-ea38-4b19-95d8-3e2bdc3a3fb1/volumes" Nov 03 20:37:16.023755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:16.023674 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx"] Nov 03 20:37:16.026005 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:16.025983 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d17e52e-d2nwx"] Nov 03 20:37:16.418878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:16.418842 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-n42qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 03 20:37:16.419051 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:16.418907 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" podUID="2e3b005c-f816-4fbd-8812-b007b7fce9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 03 20:37:17.418507 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:17.418469 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-n42qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 03 20:37:17.419032 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:17.418526 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" podUID="2e3b005c-f816-4fbd-8812-b007b7fce9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 03 20:37:17.755413 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:17.755315 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b9323b3-cc79-4b20-a4fb-d69e42e0e038" path="/var/lib/kubelet/pods/3b9323b3-cc79-4b20-a4fb-d69e42e0e038/volumes" Nov 03 20:37:18.418668 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:18.418632 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-n42qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 03 20:37:18.419082 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:18.418683 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" podUID="2e3b005c-f816-4fbd-8812-b007b7fce9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 03 20:37:19.419237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:19.419201 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-n42qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 03 20:37:19.419765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:19.419262 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" podUID="2e3b005c-f816-4fbd-8812-b007b7fce9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 03 20:37:20.418915 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:20.418880 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-n42qd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" start-of-body= Nov 03 20:37:20.419092 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:20.418946 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" podUID="2e3b005c-f816-4fbd-8812-b007b7fce9e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.253:15021/healthz/ready\": dial tcp 10.133.0.253:15021: connect: connection refused" Nov 03 20:37:21.887555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:21.887509 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:21.888320 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:21.888299 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:21.888906 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:21.888885 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-n42qd" Nov 03 20:37:25.024838 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:25.024793 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w"] Nov 03 20:37:25.028236 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:25.028206 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65a7e526-m9j2w"] Nov 03 20:37:25.756797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:25.756762 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396c4ede-c1f9-4a2b-8130-17144013bf31" path="/var/lib/kubelet/pods/396c4ede-c1f9-4a2b-8130-17144013bf31/volumes" Nov 03 20:37:28.021312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:28.021273 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm"] Nov 03 20:37:28.023896 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:28.023869 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b922a6-skwnm"] Nov 03 20:37:29.755547 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:29.755510 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e11096-489d-4ab1-9340-785ce05810ee" path="/var/lib/kubelet/pods/33e11096-489d-4ab1-9340-785ce05810ee/volumes" Nov 03 20:37:35.023132 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:35.023089 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7"] Nov 03 20:37:35.025396 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:35.025364 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-206f0544-nbbm7"] Nov 03 20:37:35.755332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:37:35.755295 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba338dc2-d4f7-45b5-ad98-a8a76121203d" path="/var/lib/kubelet/pods/ba338dc2-d4f7-45b5-ad98-a8a76121203d/volumes" Nov 03 20:37:36.145234 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:37:36.145199 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:38:06.144428 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:38:06.144373 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:38:08.260708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:38:08.260672 4457 scope.go:122] "RemoveContainer" containerID="4244b9ffdd1e958d07ebaeec1b5e4ac72be4c12aba54d0f76fcfe2a54b209128" Nov 03 20:38:08.270345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:38:08.270325 4457 scope.go:122] "RemoveContainer" containerID="8c565f054a633828b6d0fd16eea5f29ac27c70741d18ce72bbe7b947c0a1f648" Nov 03 20:38:08.279024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:38:08.279005 4457 scope.go:122] "RemoveContainer" containerID="9ed3e9089c0f6a1019b07d8ebffbb467d75199a2094370ed4ecc0205c1200b70" Nov 03 20:38:17.034491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:38:17.034413 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g"] Nov 03 20:38:17.037463 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:38:17.037419 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj"] Nov 03 20:38:17.039809 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:38:17.039779 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e8d85a-pct6g"] Nov 03 20:38:17.041813 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:38:17.041790 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ed006a1-2bhgj"] Nov 03 20:38:17.756083 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:38:17.756048 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f1debc-c299-423c-91ef-393fb9b8bdac" path="/var/lib/kubelet/pods/13f1debc-c299-423c-91ef-393fb9b8bdac/volumes" Nov 03 20:38:17.756510 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:38:17.756495 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e" path="/var/lib/kubelet/pods/a3113d6b-7df2-4bfb-b272-d0a29e1f5e6e/volumes" Nov 03 20:38:36.145081 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:38:36.145043 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:39:06.145873 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:39:06.145834 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:39:07.024848 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:07.024792 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl"] Nov 03 20:39:07.027150 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:07.027120 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-499b0549-6rctl"] Nov 03 20:39:07.756256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:07.756223 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67bd4112-42a5-485f-9a4f-279ffcda0d4e" path="/var/lib/kubelet/pods/67bd4112-42a5-485f-9a4f-279ffcda0d4e/volumes" Nov 03 20:39:08.311924 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:08.311889 4457 scope.go:122] "RemoveContainer" containerID="01b6be8012b0ce94d4e436b0152c6fbe9cc7e2247b2d6114af09a30363415efa" Nov 03 20:39:08.321005 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:08.320985 4457 scope.go:122] "RemoveContainer" containerID="83858d8a9d4e22e6f1865b806db47d3957ce58f6638447dfa572d82e4e969834" Nov 03 20:39:08.329730 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:08.329710 4457 scope.go:122] "RemoveContainer" containerID="a241caeecce6275f5facfd52904965f33032c9d153cd13273602a57cfaab6d65" Nov 03 20:39:08.337751 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:08.337722 4457 scope.go:122] "RemoveContainer" containerID="81cc399d6bbe330f89151e88555c08d70bdf9a18f52f592c4b657aced8951e5e" Nov 03 20:39:08.345645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:08.345626 4457 scope.go:122] "RemoveContainer" containerID="31c8e43d3ab3a6b4944ddcffc56fe287c05804f6d902e9e69f293c37edcc01fa" Nov 03 20:39:08.353792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:08.353774 4457 scope.go:122] "RemoveContainer" containerID="8139269e799af25211c55c4290abccd366ba0f821b2a5f60660c8b5f6c0ec737" Nov 03 20:39:08.364237 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:08.364213 4457 scope.go:122] "RemoveContainer" containerID="1ec8e144f67ae9c68c5f9e94dacccec85c72c984779f0dcef154290db59af9c4" Nov 03 20:39:08.372119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:08.372098 4457 scope.go:122] "RemoveContainer" containerID="2e979c827788d2855cfd94104f3b6c69351fbf3e609bb71d83d51dc242456283" Nov 03 20:39:08.380119 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:08.380098 4457 scope.go:122] "RemoveContainer" containerID="711c7cd70db2594ed94782f82cc81379e19d5c82ed8a456a852ae817e1b12150" Nov 03 20:39:08.387791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:08.387769 4457 scope.go:122] "RemoveContainer" containerID="8d51264cdf77f69c73cfc1c9bb608d61ac87b20191893a1d9c6455057665df13" Nov 03 20:39:08.395732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:08.395715 4457 scope.go:122] "RemoveContainer" containerID="26d2927a1652c541e80ec59fa6854c1e336b447531bc614f86d7d771d8d33f21" Nov 03 20:39:08.403556 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:08.403536 4457 scope.go:122] "RemoveContainer" containerID="781eb79cc62759db81bea5a3c36a5e0f567abdd780e111da0302553b909125ad" Nov 03 20:39:08.411457 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:08.411435 4457 scope.go:122] "RemoveContainer" containerID="92ca6cfa00b6ea30bc0e720c5f9f86629ea6b6cf7fd62585ab791b7652dc2600" Nov 03 20:39:33.024139 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:33.024106 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr"] Nov 03 20:39:33.027542 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:33.027514 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4"] Nov 03 20:39:33.030522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:33.030508 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d485fb8a-74nxr"] Nov 03 20:39:33.033480 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:33.033455 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f9c5cd1-q6fs4"] Nov 03 20:39:33.756017 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:33.755974 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf78616-ccd3-468d-aa28-bf3f4d98e66c" path="/var/lib/kubelet/pods/4cf78616-ccd3-468d-aa28-bf3f4d98e66c/volumes" Nov 03 20:39:33.756435 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:33.756418 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d43de03-9d03-4b5d-8fee-350088da28e9" path="/var/lib/kubelet/pods/8d43de03-9d03-4b5d-8fee-350088da28e9/volumes" Nov 03 20:39:36.145411 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:39:36.145348 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:39:49.025268 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:49.025225 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx"] Nov 03 20:39:49.028382 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:49.028349 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6833322-n22mx"] Nov 03 20:39:49.755889 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:49.755855 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd7bc05e-97c6-4904-9dec-aa88a2beef33" path="/var/lib/kubelet/pods/dd7bc05e-97c6-4904-9dec-aa88a2beef33/volumes" Nov 03 20:39:53.028122 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:53.028084 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps"] Nov 03 20:39:53.031032 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:53.031007 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-838f0162-2g6ps"] Nov 03 20:39:53.755706 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:39:53.755673 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0ca0910-85fc-4163-a801-3f58ea5578e1" path="/var/lib/kubelet/pods/d0ca0910-85fc-4163-a801-3f58ea5578e1/volumes" Nov 03 20:40:06.144728 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:40:06.144685 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:40:08.480510 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:08.480474 4457 scope.go:122] "RemoveContainer" containerID="b78bbca99cdfe6e5da1293e64f0389aeae8836f57c30ae21143e6bf2df03ec7d" Nov 03 20:40:08.489641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:08.489622 4457 scope.go:122] "RemoveContainer" containerID="ec45db5e71e9c0b5f0126ac1fd22bd2a6d7ef1f761e4032eeee8a26a163b3114" Nov 03 20:40:08.497305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:08.497288 4457 scope.go:122] "RemoveContainer" containerID="247a7b4603bd4d458657cb5746ece7a017eb3cba7b67089ce9f7177b7b394096" Nov 03 20:40:08.504952 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:08.504927 4457 scope.go:122] "RemoveContainer" containerID="8e3dfb7945465f895db9f2d1c26159c532fd639cce7995f37f70cde1ba7c8242" Nov 03 20:40:08.512789 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:08.512772 4457 scope.go:122] "RemoveContainer" containerID="81495a6689d358d2a7de9eb6f4124b6e4a8be31640f2e5961a37323d26ec0d9d" Nov 03 20:40:08.520530 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:08.520512 4457 scope.go:122] "RemoveContainer" containerID="fda88f3734455df6dad4aa7842d0a0dd9052845abec08483440e3a1122b61a84" Nov 03 20:40:08.528433 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:08.528414 4457 scope.go:122] "RemoveContainer" containerID="3113a3c3858a888be405ef967dce24272171dd8451e7635de5469367453d0576" Nov 03 20:40:36.145602 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:40:36.145562 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:40:39.865533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:39.865492 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-p7lrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 20:40:39.866305 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:39.866265 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-p7lrd" podUID="93f7f2c5-96a8-4ccc-ad9b-3cc648f85802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 20:40:53.160919 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.160876 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd"] Nov 03 20:40:53.161379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.161122 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" podUID="f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" containerName="discovery" containerID="cri-o://fb9ba3ad9cb48b15d2748c2a05914463e7b05f1df6744ffaab9aa0c99f8ab6ad" gracePeriod=30 Nov 03 20:40:53.426745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.426719 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:40:53.565947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.565023 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-cacerts\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-cacerts\") pod \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " Nov 03 20:40:53.565947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.565087 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-kube-api-access-hlk9s\" (UniqueName: \"kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-kube-api-access-hlk9s\") pod \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " Nov 03 20:40:53.565947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.565118 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-kubeconfig\") pod \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " Nov 03 20:40:53.565947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.565190 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-dns-cert\") pod \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " Nov 03 20:40:53.565947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.565224 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-local-certs\") pod \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " Nov 03 20:40:53.565947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.565264 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-token\" (UniqueName: \"kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-token\") pod \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " Nov 03 20:40:53.565947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.565415 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-ca-configmap\") pod \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\" (UID: \"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0\") " Nov 03 20:40:53.566936 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.566664 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-ca-configmap" pod "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" (UID: "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:40:53.569924 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.569307 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-dns-cert" pod "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" (UID: "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:53.569924 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.569663 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-cacerts" pod "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" (UID: "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:53.571253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.571207 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-kubeconfig" pod "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" (UID: "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:40:53.571253 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.571218 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-kube-api-access-hlk9s" pod "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" (UID: "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0"). InnerVolumeSpecName "kube-api-access-hlk9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:53.571453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.571434 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-token" pod "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" (UID: "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:40:53.589848 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.589800 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-local-certs" pod "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" (UID: "f2965dfe-13cd-4d50-97d1-8d48eb46d9e0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:40:53.666193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.666100 4457 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-dns-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:53.666193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.666132 4457 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-local-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:53.666193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.666145 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:53.666193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.666158 4457 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-csr-ca-configmap\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:53.666193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.666173 4457 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-cacerts\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:53.666193 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.666185 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlk9s\" (UniqueName: \"kubernetes.io/projected/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-kube-api-access-hlk9s\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:53.666609 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.666198 4457 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0-istio-kubeconfig\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:40:53.932880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.932790 4457 generic.go:356] "Generic (PLEG): container finished" podID="f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" containerID="fb9ba3ad9cb48b15d2748c2a05914463e7b05f1df6744ffaab9aa0c99f8ab6ad" exitCode=0 Nov 03 20:40:53.932880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.932842 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" Nov 03 20:40:53.933084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.932843 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" event={"ID":"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0","Type":"ContainerDied","Data":"fb9ba3ad9cb48b15d2748c2a05914463e7b05f1df6744ffaab9aa0c99f8ab6ad"} Nov 03 20:40:53.933084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.932957 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd" event={"ID":"f2965dfe-13cd-4d50-97d1-8d48eb46d9e0","Type":"ContainerDied","Data":"9b92526c24a81b3692050d21af4b5b37be21bd593cc8b764ef6a664750213e36"} Nov 03 20:40:53.933084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.932982 4457 scope.go:122] "RemoveContainer" containerID="fb9ba3ad9cb48b15d2748c2a05914463e7b05f1df6744ffaab9aa0c99f8ab6ad" Nov 03 20:40:53.943470 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.943450 4457 scope.go:122] "RemoveContainer" containerID="fb9ba3ad9cb48b15d2748c2a05914463e7b05f1df6744ffaab9aa0c99f8ab6ad" Nov 03 20:40:53.943761 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:40:53.943737 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9ba3ad9cb48b15d2748c2a05914463e7b05f1df6744ffaab9aa0c99f8ab6ad\": container with ID starting with fb9ba3ad9cb48b15d2748c2a05914463e7b05f1df6744ffaab9aa0c99f8ab6ad not found: ID does not exist" containerID="fb9ba3ad9cb48b15d2748c2a05914463e7b05f1df6744ffaab9aa0c99f8ab6ad" Nov 03 20:40:53.943811 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.943770 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9ba3ad9cb48b15d2748c2a05914463e7b05f1df6744ffaab9aa0c99f8ab6ad"} err="failed to get container status \"fb9ba3ad9cb48b15d2748c2a05914463e7b05f1df6744ffaab9aa0c99f8ab6ad\": rpc error: code = NotFound desc = could not find container \"fb9ba3ad9cb48b15d2748c2a05914463e7b05f1df6744ffaab9aa0c99f8ab6ad\": container with ID starting with fb9ba3ad9cb48b15d2748c2a05914463e7b05f1df6744ffaab9aa0c99f8ab6ad not found: ID does not exist" Nov 03 20:40:53.955805 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.955773 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd"] Nov 03 20:40:53.960802 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:53.960776 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lscmd"] Nov 03 20:40:55.756659 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:40:55.756616 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" path="/var/lib/kubelet/pods/f2965dfe-13cd-4d50-97d1-8d48eb46d9e0/volumes" Nov 03 20:41:06.154726 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:41:06.154686 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:41:08.572708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:41:08.572673 4457 scope.go:122] "RemoveContainer" containerID="62d0cd2f957ecc2251128e5a2a3694526f10ec73146b7c5984c3acf3396c1ca3" Nov 03 20:41:08.581651 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:41:08.581627 4457 scope.go:122] "RemoveContainer" containerID="25b9514eb6e2ee6c474620e9ae8a3c7b0cccd71a896da13c405394da2ebe1000" Nov 03 20:41:08.590469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:41:08.590448 4457 scope.go:122] "RemoveContainer" containerID="531bc7bc8c660b3381ae0c360063fa9606548f657d858f2b3c81ece425f9657f" Nov 03 20:41:08.598888 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:41:08.598868 4457 scope.go:122] "RemoveContainer" containerID="eb13ca2c11e5385d03f1d7d6b3d520739769703b1c671790c220ce01f4918b83" Nov 03 20:41:08.606498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:41:08.606472 4457 scope.go:122] "RemoveContainer" containerID="7a83d455db423f0db1ce770ff39b1a5339e69ffa2bbe9eb9dcc3dffab1805b3f" Nov 03 20:41:08.614167 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:41:08.614148 4457 scope.go:122] "RemoveContainer" containerID="44dcc0e7e47fd62d7897a4c25c8f0178678281c2ee10b83a3a4b738f4dec7ae1" Nov 03 20:41:08.621571 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:41:08.621550 4457 scope.go:122] "RemoveContainer" containerID="f922c9e067f054082a678628002a734b8f6498c635ee2dfbd3e29d48aaa70099" Nov 03 20:41:08.629645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:41:08.629620 4457 scope.go:122] "RemoveContainer" containerID="adbd42d72873a0730b4f71ef4302ebe69fa659c4b819ae0ece627538c395723a" Nov 03 20:41:36.239001 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:41:36.237297 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:42:02.911691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:42:02.911659 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:42:02.915307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:42:02.915278 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:42:02.926862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:42:02.926836 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:42:02.930143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:42:02.930120 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:42:06.145738 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:42:06.145696 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:42:36.145717 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:42:36.145674 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:43:06.143754 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:43:06.143715 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:43:36.146029 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:43:36.145917 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:44:06.144161 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:44:06.144112 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:44:36.144203 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:44:36.144106 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:45:06.145425 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:45:06.145372 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:45:36.145464 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:45:36.145423 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:46:06.145200 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:46:06.145158 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:46:36.145676 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:46:36.145640 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:47:02.949668 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:02.949633 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:47:02.953683 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:02.953660 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:47:02.966340 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:02.966314 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:47:02.970144 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:02.970119 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:47:06.146162 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:47:06.146116 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:47:29.566506 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.566454 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt"] Nov 03 20:47:29.567037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.567020 4457 cpu_manager.go:414] "containerMap: removing container" podUID="f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" containerName="discovery" Nov 03 20:47:29.567091 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.567041 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" containerName="discovery" Nov 03 20:47:29.567196 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.567184 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2965dfe-13cd-4d50-97d1-8d48eb46d9e0" containerName="discovery" Nov 03 20:47:29.567242 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.567204 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d097968-5008-4658-a534-fe81ba625490" containerName="istio-proxy" Nov 03 20:47:29.570882 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.570851 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.575962 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.575930 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-e95cf7-openshift-default-dockercfg-gwfs6\"" Nov 03 20:47:29.603256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.597457 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt"] Nov 03 20:47:29.603256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.603113 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f41d872-4ee5-45b3-bfab-653323283361-istio-podinfo\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.603256 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.603166 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-envoy\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.603588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.603268 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6szv\" (UniqueName: \"kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-kube-api-access-x6szv\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.603588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.603319 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-istio-token\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.603588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.603398 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f41d872-4ee5-45b3-bfab-653323283361-istiod-ca-cert\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.603588 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.603479 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-certs\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.603808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.603586 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-data\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.603808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.603617 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-socket\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.603808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.603662 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-credential-socket\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.705003 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.704962 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-credential-socket\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.705221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.705016 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f41d872-4ee5-45b3-bfab-653323283361-istio-podinfo\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.705221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.705033 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-envoy\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.705221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.705179 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6szv\" (UniqueName: \"kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-kube-api-access-x6szv\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.705453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.705221 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-istio-token\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.705453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.705323 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f41d872-4ee5-45b3-bfab-653323283361-istiod-ca-cert\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.705453 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.705382 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-certs\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.705641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.705471 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-data\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.705641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.705501 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-socket\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.705641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.705546 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-credential-socket\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.705866 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.705839 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-certs\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.705967 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.705905 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-data\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.705967 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.705923 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-socket\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.706282 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.706259 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f41d872-4ee5-45b3-bfab-653323283361-istiod-ca-cert\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.707885 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.707860 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f41d872-4ee5-45b3-bfab-653323283361-istio-podinfo\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.708184 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.708157 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-envoy\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.717548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.717518 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-istio-token\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.718059 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.718037 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6szv\" (UniqueName: \"kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-kube-api-access-x6szv\") pod \"e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:29.899947 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:29.899851 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:30.185043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:30.185009 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt"] Nov 03 20:47:30.187212 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:47:30.187182 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f41d872_4ee5_45b3_bfab_653323283361.slice/crio-ebc15936a0bab785756ae750e565440be8d7fcd09a3d32886c3a6e43c360babf WatchSource:0}: Error finding container ebc15936a0bab785756ae750e565440be8d7fcd09a3d32886c3a6e43c360babf: Status 404 returned error can't find the container with id ebc15936a0bab785756ae750e565440be8d7fcd09a3d32886c3a6e43c360babf Nov 03 20:47:30.189468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:30.189438 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:30.189537 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:30.189506 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:30.189576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:30.189537 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:31.048642 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:31.048592 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" event={"ID":"8f41d872-4ee5-45b3-bfab-653323283361","Type":"ContainerStarted","Data":"b987233571128afcf0ccf1b8b5ce9e5df475f4887d0326bb5cf7465816f19db0"} Nov 03 20:47:31.049153 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:31.048659 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" event={"ID":"8f41d872-4ee5-45b3-bfab-653323283361","Type":"ContainerStarted","Data":"ebc15936a0bab785756ae750e565440be8d7fcd09a3d32886c3a6e43c360babf"} Nov 03 20:47:31.076085 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:31.076013 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" podStartSLOduration=2.075993941 podStartE2EDuration="2.075993941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:31.073917488 +0000 UTC m=+3029.449514247" watchObservedRunningTime="2026-11-03 20:47:31.075993941 +0000 UTC m=+3029.451590690" Nov 03 20:47:31.900815 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:31.900780 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:31.905704 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:31.905680 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:32.054141 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:32.054108 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:32.055280 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:32.055260 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:36.147083 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:47:36.147042 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:47:43.831202 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:43.831140 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk"] Nov 03 20:47:43.836667 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:43.836640 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:43.851423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:43.847752 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk"] Nov 03 20:47:43.941657 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:43.941604 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:43.941657 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:43.941648 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:43.941941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:43.941717 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad84ee89-c964-4229-be80-bb309afb1255-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:43.941941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:43.941776 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad84ee89-c964-4229-be80-bb309afb1255-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:43.941941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:43.941854 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgrwg\" (UniqueName: \"kubernetes.io/projected/ad84ee89-c964-4229-be80-bb309afb1255-kube-api-access-sgrwg\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:43.941941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:43.941887 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:43.941941 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:43.941938 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:43.942234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:43.941969 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:43.942234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:43.941989 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad84ee89-c964-4229-be80-bb309afb1255-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.042922 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad84ee89-c964-4229-be80-bb309afb1255-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.042982 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgrwg\" (UniqueName: \"kubernetes.io/projected/ad84ee89-c964-4229-be80-bb309afb1255-kube-api-access-sgrwg\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.043012 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043030 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.043031 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.043062 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.043079 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad84ee89-c964-4229-be80-bb309afb1255-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.043101 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.043130 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043407 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.043177 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad84ee89-c964-4229-be80-bb309afb1255-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.043544 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.043562 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043830 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.043724 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043830 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.043762 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.043973 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.043950 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad84ee89-c964-4229-be80-bb309afb1255-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.045551 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.045531 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad84ee89-c964-4229-be80-bb309afb1255-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.045782 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.045762 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad84ee89-c964-4229-be80-bb309afb1255-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.055621 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.055591 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad84ee89-c964-4229-be80-bb309afb1255-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.055954 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.055929 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgrwg\" (UniqueName: \"kubernetes.io/projected/ad84ee89-c964-4229-be80-bb309afb1255-kube-api-access-sgrwg\") pod \"maas-default-gateway-openshift-default-77f556d647-g6znk\" (UID: \"ad84ee89-c964-4229-be80-bb309afb1255\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.155991 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.155948 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:44.444296 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.444263 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk"] Nov 03 20:47:44.446333 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:47:44.446298 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad84ee89_c964_4229_be80_bb309afb1255.slice/crio-67e7e19b101fdb2bf53d4a3a5f749b3d1c247d12d54d1a4f9aa8657e9c5ced3a WatchSource:0}: Error finding container 67e7e19b101fdb2bf53d4a3a5f749b3d1c247d12d54d1a4f9aa8657e9c5ced3a: Status 404 returned error can't find the container with id 67e7e19b101fdb2bf53d4a3a5f749b3d1c247d12d54d1a4f9aa8657e9c5ced3a Nov 03 20:47:44.448846 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.448808 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:44.448962 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.448879 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:44.448962 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:44.448909 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:45.124829 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:45.124788 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" event={"ID":"ad84ee89-c964-4229-be80-bb309afb1255","Type":"ContainerStarted","Data":"6dd2cd5a2bfcad8d1c766116a8257bfa83599f3c8252459b6f170f7dc5712cae"} Nov 03 20:47:45.125334 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:45.124852 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" event={"ID":"ad84ee89-c964-4229-be80-bb309afb1255","Type":"ContainerStarted","Data":"67e7e19b101fdb2bf53d4a3a5f749b3d1c247d12d54d1a4f9aa8657e9c5ced3a"} Nov 03 20:47:45.150069 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:45.150011 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" podStartSLOduration=2.149997069 podStartE2EDuration="2.149997069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:45.148978674 +0000 UTC m=+3043.524575422" watchObservedRunningTime="2026-11-03 20:47:45.149997069 +0000 UTC m=+3043.525593816" Nov 03 20:47:45.172411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:45.166480 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:45.182890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:45.182621 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Nov 03 20:47:45.182890 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:45.182703 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" podUID="ad84ee89-c964-4229-be80-bb309afb1255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Nov 03 20:47:46.163821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:46.163778 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Nov 03 20:47:46.164476 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:46.164438 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" podUID="ad84ee89-c964-4229-be80-bb309afb1255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Nov 03 20:47:47.101519 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:47.101484 4457 patch_prober.go:29] interesting pod/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": read tcp 10.133.0.2:60544->10.133.0.254:15021: read: connection reset by peer" start-of-body= Nov 03 20:47:47.101702 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:47.101550 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": read tcp 10.133.0.2:60544->10.133.0.254:15021: read: connection reset by peer" Nov 03 20:47:47.142437 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:47.142403 4457 generic.go:356] "Generic (PLEG): container finished" podID="8f41d872-4ee5-45b3-bfab-653323283361" containerID="b987233571128afcf0ccf1b8b5ce9e5df475f4887d0326bb5cf7465816f19db0" exitCode=0 Nov 03 20:47:47.142437 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:47.142420 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" event={"ID":"8f41d872-4ee5-45b3-bfab-653323283361","Type":"ContainerDied","Data":"b987233571128afcf0ccf1b8b5ce9e5df475f4887d0326bb5cf7465816f19db0"} Nov 03 20:47:47.143011 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:47.142993 4457 scope.go:122] "RemoveContainer" containerID="b987233571128afcf0ccf1b8b5ce9e5df475f4887d0326bb5cf7465816f19db0" Nov 03 20:47:47.144186 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:47.144157 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:47.144246 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:47.144224 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:47.144286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:47.144255 4457 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Nov 03 20:47:47.186876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:47.186841 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Nov 03 20:47:47.187344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:47.186907 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" podUID="ad84ee89-c964-4229-be80-bb309afb1255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Nov 03 20:47:48.149310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:48.149264 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" event={"ID":"8f41d872-4ee5-45b3-bfab-653323283361","Type":"ContainerStarted","Data":"a8ca7831f44451fd7db8cb60510a9e7f10cb6a9084c706842a30a2316f3a66ed"} Nov 03 20:47:48.171593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:48.171534 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Nov 03 20:47:48.172099 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:48.172059 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" podUID="ad84ee89-c964-4229-be80-bb309afb1255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Nov 03 20:47:48.903874 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:48.903837 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:48.904674 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:48.904567 4457 patch_prober.go:29] interesting pod/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:47:48.904876 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:48.904846 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Nov 03 20:47:49.159892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:49.159791 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Nov 03 20:47:49.160183 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:49.160151 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" podUID="ad84ee89-c964-4229-be80-bb309afb1255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Nov 03 20:47:49.905837 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:49.905784 4457 patch_prober.go:29] interesting pod/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:47:49.906372 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:49.905858 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Nov 03 20:47:50.164467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.162531 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Nov 03 20:47:50.164467 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.162606 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" podUID="ad84ee89-c964-4229-be80-bb309afb1255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Nov 03 20:47:50.226547 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.224360 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8"] Nov 03 20:47:50.227465 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.227444 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.231322 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.231295 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e95cf7\"/\"openshift-service-ca.crt\"" Nov 03 20:47:50.232239 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.232223 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e95cf7\"/\"kube-root-ca.crt\"" Nov 03 20:47:50.238184 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.238155 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8"] Nov 03 20:47:50.289507 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.289305 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e95cf7\"/\"default-dockercfg-bh5zv\"" Nov 03 20:47:50.289507 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.289455 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e95cf7\"/\"simulator-single-main-e95cf7-kserve-self-signed-certs\"" Nov 03 20:47:50.335356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.334909 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d11bc51d-2a96-4de7-bfe2-485734b2403b-tls-certs\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.335356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.334989 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26w86\" (UniqueName: \"kubernetes.io/projected/d11bc51d-2a96-4de7-bfe2-485734b2403b-kube-api-access-26w86\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.335356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.335081 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-dshm\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.335356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.335111 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-home\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.335356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.335164 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-tmp-dir\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.335356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.335193 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-model-cache\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.436178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.436077 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-dshm\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.436178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.436117 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-home\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.436178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.436161 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-tmp-dir\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.436178 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.436179 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-model-cache\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.436566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.436217 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d11bc51d-2a96-4de7-bfe2-485734b2403b-tls-certs\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.436566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.436250 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26w86\" (UniqueName: \"kubernetes.io/projected/d11bc51d-2a96-4de7-bfe2-485734b2403b-kube-api-access-26w86\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.437148 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.437124 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-home\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.437492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.437476 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-model-cache\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.437855 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.437829 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-tmp-dir\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.439146 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.439123 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-dshm\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.441062 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.441039 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d11bc51d-2a96-4de7-bfe2-485734b2403b-tls-certs\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.449154 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.449135 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26w86\" (UniqueName: \"kubernetes.io/projected/d11bc51d-2a96-4de7-bfe2-485734b2403b-kube-api-access-26w86\") pod \"simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.551952 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.551916 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:50.880412 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.880052 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8"] Nov 03 20:47:50.900494 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.900457 4457 patch_prober.go:29] interesting pod/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:47:50.900613 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:50.900515 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Nov 03 20:47:51.157339 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:51.156806 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Nov 03 20:47:51.157339 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:51.156874 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" podUID="ad84ee89-c964-4229-be80-bb309afb1255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Nov 03 20:47:51.179411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:51.179250 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" event={"ID":"d11bc51d-2a96-4de7-bfe2-485734b2403b","Type":"ContainerStarted","Data":"f3dd63b355e99a4cb2b1c205ecdf56aec2d78c8e458a0a4c55e17150b5e7aa4f"} Nov 03 20:47:51.179411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:51.179311 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" event={"ID":"d11bc51d-2a96-4de7-bfe2-485734b2403b","Type":"ContainerStarted","Data":"f3cebb50fec62f22b38fc4aa1d9caae354baa968381b6525e4c6b144a9482f02"} Nov 03 20:47:51.552978 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:51.552941 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:47:51.901037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:51.900939 4457 patch_prober.go:29] interesting pod/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:47:51.901037 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:51.901009 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Nov 03 20:47:52.165474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:52.163816 4457 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-g6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" start-of-body= Nov 03 20:47:52.165474 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:52.163880 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" podUID="ad84ee89-c964-4229-be80-bb309afb1255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.255:15021/healthz/ready\": dial tcp 10.133.0.255:15021: connect: connection refused" Nov 03 20:47:52.904411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:52.903782 4457 patch_prober.go:29] interesting pod/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Nov 03 20:47:52.904411 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:52.903848 4457 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Nov 03 20:47:53.489312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:53.489277 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:53.490110 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:53.490075 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:53.491465 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:53.491443 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-g6znk" Nov 03 20:47:53.514695 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:53.514627 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" podStartSLOduration=3.514610325 podStartE2EDuration="3.514610325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:47:51.203782237 +0000 UTC m=+3049.579378985" watchObservedRunningTime="2026-11-03 20:47:53.514610325 +0000 UTC m=+3051.890207073" Nov 03 20:47:53.911283 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:53.911247 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:53.913041 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:53.912566 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:47:53.916787 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:47:53.916760 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:48:00.552569 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:00.552528 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:48:00.565169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:00.565134 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:48:01.253718 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:01.253683 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:48:06.144814 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:48:06.144774 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:48:20.650739 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.650705 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j"] Nov 03 20:48:20.654432 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.654413 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" Nov 03 20:48:20.661884 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.661358 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:48:20.662036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.662015 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:48:20.663157 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.663132 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j"] Nov 03 20:48:20.758211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.758173 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d90d32e-ws54j\" (UID: \"bc35d6f0-1471-4daf-9549-ccc179797c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" Nov 03 20:48:20.758211 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.758212 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26vkz\" (UniqueName: \"kubernetes.io/projected/bc35d6f0-1471-4daf-9549-ccc179797c93-kube-api-access-26vkz\") pod \"maas-api-revoke-sub-8d90d32e-ws54j\" (UID: \"bc35d6f0-1471-4daf-9549-ccc179797c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" Nov 03 20:48:20.758507 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.758422 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d90d32e-ws54j\" (UID: \"bc35d6f0-1471-4daf-9549-ccc179797c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" Nov 03 20:48:20.859562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.859527 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d90d32e-ws54j\" (UID: \"bc35d6f0-1471-4daf-9549-ccc179797c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" Nov 03 20:48:20.859562 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.859564 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26vkz\" (UniqueName: \"kubernetes.io/projected/bc35d6f0-1471-4daf-9549-ccc179797c93-kube-api-access-26vkz\") pod \"maas-api-revoke-sub-8d90d32e-ws54j\" (UID: \"bc35d6f0-1471-4daf-9549-ccc179797c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" Nov 03 20:48:20.859798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.859628 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d90d32e-ws54j\" (UID: \"bc35d6f0-1471-4daf-9549-ccc179797c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" Nov 03 20:48:20.860298 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.860277 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d90d32e-ws54j\" (UID: \"bc35d6f0-1471-4daf-9549-ccc179797c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" Nov 03 20:48:20.861878 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.861853 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d90d32e-ws54j\" (UID: \"bc35d6f0-1471-4daf-9549-ccc179797c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" Nov 03 20:48:20.871634 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.871614 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26vkz\" (UniqueName: \"kubernetes.io/projected/bc35d6f0-1471-4daf-9549-ccc179797c93-kube-api-access-26vkz\") pod \"maas-api-revoke-sub-8d90d32e-ws54j\" (UID: \"bc35d6f0-1471-4daf-9549-ccc179797c93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" Nov 03 20:48:20.968332 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:20.968259 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" Nov 03 20:48:21.242637 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:21.242609 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j"] Nov 03 20:48:21.244914 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:48:21.244884 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc35d6f0_1471_4daf_9549_ccc179797c93.slice/crio-73767ea033354cd1c8f09ec380f61622e5e7e3c9df18f865d01186c28cea1a08 WatchSource:0}: Error finding container 73767ea033354cd1c8f09ec380f61622e5e7e3c9df18f865d01186c28cea1a08: Status 404 returned error can't find the container with id 73767ea033354cd1c8f09ec380f61622e5e7e3c9df18f865d01186c28cea1a08 Nov 03 20:48:21.365380 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:21.365344 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" event={"ID":"bc35d6f0-1471-4daf-9549-ccc179797c93","Type":"ContainerStarted","Data":"73767ea033354cd1c8f09ec380f61622e5e7e3c9df18f865d01186c28cea1a08"} Nov 03 20:48:22.372448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:22.372345 4457 generic.go:356] "Generic (PLEG): container finished" podID="bc35d6f0-1471-4daf-9549-ccc179797c93" containerID="78f007bd15c65537d98b509312a35d2cc91295567fe28d766667316ed03b4ff6" exitCode=0 Nov 03 20:48:22.372448 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:22.372428 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" event={"ID":"bc35d6f0-1471-4daf-9549-ccc179797c93","Type":"ContainerDied","Data":"78f007bd15c65537d98b509312a35d2cc91295567fe28d766667316ed03b4ff6"} Nov 03 20:48:23.506129 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:23.506104 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" Nov 03 20:48:23.585567 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:23.585534 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc35d6f0-1471-4daf-9549-ccc179797c93-kube-api-access-26vkz\" (UniqueName: \"kubernetes.io/projected/bc35d6f0-1471-4daf-9549-ccc179797c93-kube-api-access-26vkz\") pod \"bc35d6f0-1471-4daf-9549-ccc179797c93\" (UID: \"bc35d6f0-1471-4daf-9549-ccc179797c93\") " Nov 03 20:48:23.585735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:23.585619 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-xks-ca\") pod \"bc35d6f0-1471-4daf-9549-ccc179797c93\" (UID: \"bc35d6f0-1471-4daf-9549-ccc179797c93\") " Nov 03 20:48:23.585735 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:23.585655 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-openshift-ca\") pod \"bc35d6f0-1471-4daf-9549-ccc179797c93\" (UID: \"bc35d6f0-1471-4daf-9549-ccc179797c93\") " Nov 03 20:48:23.586084 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:23.586059 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-openshift-ca" pod "bc35d6f0-1471-4daf-9549-ccc179797c93" (UID: "bc35d6f0-1471-4daf-9549-ccc179797c93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:23.587658 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:23.587627 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-xks-ca" pod "bc35d6f0-1471-4daf-9549-ccc179797c93" (UID: "bc35d6f0-1471-4daf-9549-ccc179797c93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:23.587658 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:23.587626 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc35d6f0-1471-4daf-9549-ccc179797c93-kube-api-access-26vkz" pod "bc35d6f0-1471-4daf-9549-ccc179797c93" (UID: "bc35d6f0-1471-4daf-9549-ccc179797c93"). InnerVolumeSpecName "kube-api-access-26vkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:23.687348 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:23.687259 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26vkz\" (UniqueName: \"kubernetes.io/projected/bc35d6f0-1471-4daf-9549-ccc179797c93-kube-api-access-26vkz\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:23.687348 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:23.687293 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:23.687348 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:23.687302 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc35d6f0-1471-4daf-9549-ccc179797c93-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:24.384240 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:24.384202 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" event={"ID":"bc35d6f0-1471-4daf-9549-ccc179797c93","Type":"ContainerDied","Data":"73767ea033354cd1c8f09ec380f61622e5e7e3c9df18f865d01186c28cea1a08"} Nov 03 20:48:24.384466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:24.384284 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73767ea033354cd1c8f09ec380f61622e5e7e3c9df18f865d01186c28cea1a08" Nov 03 20:48:24.384466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:24.384213 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j" Nov 03 20:48:31.688755 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.688720 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w"] Nov 03 20:48:31.689244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.689134 4457 cpu_manager.go:414] "containerMap: removing container" podUID="bc35d6f0-1471-4daf-9549-ccc179797c93" containerName="revoke-keys" Nov 03 20:48:31.689244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.689146 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc35d6f0-1471-4daf-9549-ccc179797c93" containerName="revoke-keys" Nov 03 20:48:31.689244 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.689214 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc35d6f0-1471-4daf-9549-ccc179797c93" containerName="revoke-keys" Nov 03 20:48:31.692663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.692646 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" Nov 03 20:48:31.697843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.697820 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:48:31.699007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.698972 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:48:31.703739 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.703713 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w"] Nov 03 20:48:31.767226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.767190 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w\" (UID: \"d561a9c5-1af9-447b-826b-5d0d0785404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" Nov 03 20:48:31.767440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.767252 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w\" (UID: \"d561a9c5-1af9-447b-826b-5d0d0785404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" Nov 03 20:48:31.767533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.767434 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2zwp\" (UniqueName: \"kubernetes.io/projected/d561a9c5-1af9-447b-826b-5d0d0785404f-kube-api-access-c2zwp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w\" (UID: \"d561a9c5-1af9-447b-826b-5d0d0785404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" Nov 03 20:48:31.868501 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.868463 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w\" (UID: \"d561a9c5-1af9-447b-826b-5d0d0785404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" Nov 03 20:48:31.868710 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.868583 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2zwp\" (UniqueName: \"kubernetes.io/projected/d561a9c5-1af9-447b-826b-5d0d0785404f-kube-api-access-c2zwp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w\" (UID: \"d561a9c5-1af9-447b-826b-5d0d0785404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" Nov 03 20:48:31.868710 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.868675 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w\" (UID: \"d561a9c5-1af9-447b-826b-5d0d0785404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" Nov 03 20:48:31.869164 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.869142 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w\" (UID: \"d561a9c5-1af9-447b-826b-5d0d0785404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" Nov 03 20:48:31.871081 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.871060 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w\" (UID: \"d561a9c5-1af9-447b-826b-5d0d0785404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" Nov 03 20:48:31.880310 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:31.880288 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2zwp\" (UniqueName: \"kubernetes.io/projected/d561a9c5-1af9-447b-826b-5d0d0785404f-kube-api-access-c2zwp\") pod \"maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w\" (UID: \"d561a9c5-1af9-447b-826b-5d0d0785404f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" Nov 03 20:48:32.007163 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:32.007085 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" Nov 03 20:48:32.317996 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:32.317968 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w"] Nov 03 20:48:32.319696 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:48:32.319671 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd561a9c5_1af9_447b_826b_5d0d0785404f.slice/crio-cf4e186e3e05ce86fb54c95db35077899e4f63d71eed172b20da95ecb2b58dfc WatchSource:0}: Error finding container cf4e186e3e05ce86fb54c95db35077899e4f63d71eed172b20da95ecb2b58dfc: Status 404 returned error can't find the container with id cf4e186e3e05ce86fb54c95db35077899e4f63d71eed172b20da95ecb2b58dfc Nov 03 20:48:32.430753 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:32.430718 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" event={"ID":"d561a9c5-1af9-447b-826b-5d0d0785404f","Type":"ContainerStarted","Data":"cf4e186e3e05ce86fb54c95db35077899e4f63d71eed172b20da95ecb2b58dfc"} Nov 03 20:48:33.438614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:33.438528 4457 generic.go:356] "Generic (PLEG): container finished" podID="d561a9c5-1af9-447b-826b-5d0d0785404f" containerID="57fa138312f17c4dc480954246f688793984e0ef27cbb10b6b53852e5c790b61" exitCode=0 Nov 03 20:48:33.438614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:33.438596 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" event={"ID":"d561a9c5-1af9-447b-826b-5d0d0785404f","Type":"ContainerDied","Data":"57fa138312f17c4dc480954246f688793984e0ef27cbb10b6b53852e5c790b61"} Nov 03 20:48:34.580614 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:34.580582 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" Nov 03 20:48:34.697354 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:34.697276 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-openshift-ca\") pod \"d561a9c5-1af9-447b-826b-5d0d0785404f\" (UID: \"d561a9c5-1af9-447b-826b-5d0d0785404f\") " Nov 03 20:48:34.697545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:34.697414 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d561a9c5-1af9-447b-826b-5d0d0785404f-kube-api-access-c2zwp\" (UniqueName: \"kubernetes.io/projected/d561a9c5-1af9-447b-826b-5d0d0785404f-kube-api-access-c2zwp\") pod \"d561a9c5-1af9-447b-826b-5d0d0785404f\" (UID: \"d561a9c5-1af9-447b-826b-5d0d0785404f\") " Nov 03 20:48:34.697545 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:34.697465 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-xks-ca\") pod \"d561a9c5-1af9-447b-826b-5d0d0785404f\" (UID: \"d561a9c5-1af9-447b-826b-5d0d0785404f\") " Nov 03 20:48:34.697676 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:34.697650 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-openshift-ca" pod "d561a9c5-1af9-447b-826b-5d0d0785404f" (UID: "d561a9c5-1af9-447b-826b-5d0d0785404f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:34.697741 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:34.697714 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:34.699797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:34.699763 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d561a9c5-1af9-447b-826b-5d0d0785404f-kube-api-access-c2zwp" pod "d561a9c5-1af9-447b-826b-5d0d0785404f" (UID: "d561a9c5-1af9-447b-826b-5d0d0785404f"). InnerVolumeSpecName "kube-api-access-c2zwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:34.699963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:34.699829 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-xks-ca" pod "d561a9c5-1af9-447b-826b-5d0d0785404f" (UID: "d561a9c5-1af9-447b-826b-5d0d0785404f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:34.798792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:34.798752 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2zwp\" (UniqueName: \"kubernetes.io/projected/d561a9c5-1af9-447b-826b-5d0d0785404f-kube-api-access-c2zwp\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:34.798792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:34.798791 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d561a9c5-1af9-447b-826b-5d0d0785404f-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:35.450495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:35.450454 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" event={"ID":"d561a9c5-1af9-447b-826b-5d0d0785404f","Type":"ContainerDied","Data":"cf4e186e3e05ce86fb54c95db35077899e4f63d71eed172b20da95ecb2b58dfc"} Nov 03 20:48:35.450495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:35.450489 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w" Nov 03 20:48:35.450719 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:35.450518 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf4e186e3e05ce86fb54c95db35077899e4f63d71eed172b20da95ecb2b58dfc" Nov 03 20:48:36.145131 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:48:36.145086 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:48:36.494560 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:36.494458 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8"] Nov 03 20:48:42.632844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:42.632810 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w"] Nov 03 20:48:42.635869 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:42.635847 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-e95cf7-f9l7w"] Nov 03 20:48:43.755370 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:43.755338 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d561a9c5-1af9-447b-826b-5d0d0785404f" path="/var/lib/kubelet/pods/d561a9c5-1af9-447b-826b-5d0d0785404f/volumes" Nov 03 20:48:49.337998 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:49.337958 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt"] Nov 03 20:48:49.338496 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:49.338222 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" containerID="cri-o://a8ca7831f44451fd7db8cb60510a9e7f10cb6a9084c706842a30a2316f3a66ed" gracePeriod=30 Nov 03 20:48:51.514254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.514210 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" podUID="d11bc51d-2a96-4de7-bfe2-485734b2403b" containerName="main" containerID="cri-o://f3dd63b355e99a4cb2b1c205ecdf56aec2d78c8e458a0a4c55e17150b5e7aa4f" gracePeriod=45 Nov 03 20:48:51.759421 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.759379 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:48:51.874492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.874453 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d11bc51d-2a96-4de7-bfe2-485734b2403b-tls-certs\" (UniqueName: \"kubernetes.io/secret/d11bc51d-2a96-4de7-bfe2-485734b2403b-tls-certs\") pod \"d11bc51d-2a96-4de7-bfe2-485734b2403b\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " Nov 03 20:48:51.874681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.874547 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-dshm\") pod \"d11bc51d-2a96-4de7-bfe2-485734b2403b\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " Nov 03 20:48:51.874681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.874586 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-tmp-dir\") pod \"d11bc51d-2a96-4de7-bfe2-485734b2403b\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " Nov 03 20:48:51.874681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.874605 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11bc51d-2a96-4de7-bfe2-485734b2403b-kube-api-access-26w86\" (UniqueName: \"kubernetes.io/projected/d11bc51d-2a96-4de7-bfe2-485734b2403b-kube-api-access-26w86\") pod \"d11bc51d-2a96-4de7-bfe2-485734b2403b\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " Nov 03 20:48:51.874681 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.874627 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-model-cache\") pod \"d11bc51d-2a96-4de7-bfe2-485734b2403b\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " Nov 03 20:48:51.874953 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.874692 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-home\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-home\") pod \"d11bc51d-2a96-4de7-bfe2-485734b2403b\" (UID: \"d11bc51d-2a96-4de7-bfe2-485734b2403b\") " Nov 03 20:48:51.874953 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.874889 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-tmp-dir" pod "d11bc51d-2a96-4de7-bfe2-485734b2403b" (UID: "d11bc51d-2a96-4de7-bfe2-485734b2403b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:48:51.875058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.874950 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-model-cache" pod "d11bc51d-2a96-4de7-bfe2-485734b2403b" (UID: "d11bc51d-2a96-4de7-bfe2-485734b2403b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:48:51.875164 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.875143 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:51.875223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.875169 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:51.875223 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.875169 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-home" pod "d11bc51d-2a96-4de7-bfe2-485734b2403b" (UID: "d11bc51d-2a96-4de7-bfe2-485734b2403b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:48:51.876803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.876777 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d11bc51d-2a96-4de7-bfe2-485734b2403b-tls-certs" pod "d11bc51d-2a96-4de7-bfe2-485734b2403b" (UID: "d11bc51d-2a96-4de7-bfe2-485734b2403b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:48:51.876935 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.876920 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-dshm" pod "d11bc51d-2a96-4de7-bfe2-485734b2403b" (UID: "d11bc51d-2a96-4de7-bfe2-485734b2403b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:48:51.877007 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.876938 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11bc51d-2a96-4de7-bfe2-485734b2403b-kube-api-access-26w86" pod "d11bc51d-2a96-4de7-bfe2-485734b2403b" (UID: "d11bc51d-2a96-4de7-bfe2-485734b2403b"). InnerVolumeSpecName "kube-api-access-26w86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:51.976043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.976006 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:51.976043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.976039 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26w86\" (UniqueName: \"kubernetes.io/projected/d11bc51d-2a96-4de7-bfe2-485734b2403b-kube-api-access-26w86\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:51.976043 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.976050 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d11bc51d-2a96-4de7-bfe2-485734b2403b-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:51.976299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:51.976059 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d11bc51d-2a96-4de7-bfe2-485734b2403b-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:52.544489 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:52.544456 4457 generic.go:356] "Generic (PLEG): container finished" podID="d11bc51d-2a96-4de7-bfe2-485734b2403b" containerID="f3dd63b355e99a4cb2b1c205ecdf56aec2d78c8e458a0a4c55e17150b5e7aa4f" exitCode=0 Nov 03 20:48:52.544932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:52.544511 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" Nov 03 20:48:52.544932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:52.544528 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" event={"ID":"d11bc51d-2a96-4de7-bfe2-485734b2403b","Type":"ContainerDied","Data":"f3dd63b355e99a4cb2b1c205ecdf56aec2d78c8e458a0a4c55e17150b5e7aa4f"} Nov 03 20:48:52.544932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:52.544564 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8" event={"ID":"d11bc51d-2a96-4de7-bfe2-485734b2403b","Type":"ContainerDied","Data":"f3cebb50fec62f22b38fc4aa1d9caae354baa968381b6525e4c6b144a9482f02"} Nov 03 20:48:52.544932 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:52.544583 4457 scope.go:122] "RemoveContainer" containerID="f3dd63b355e99a4cb2b1c205ecdf56aec2d78c8e458a0a4c55e17150b5e7aa4f" Nov 03 20:48:52.555466 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:52.555449 4457 scope.go:122] "RemoveContainer" containerID="f3dd63b355e99a4cb2b1c205ecdf56aec2d78c8e458a0a4c55e17150b5e7aa4f" Nov 03 20:48:52.555716 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:48:52.555695 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3dd63b355e99a4cb2b1c205ecdf56aec2d78c8e458a0a4c55e17150b5e7aa4f\": container with ID starting with f3dd63b355e99a4cb2b1c205ecdf56aec2d78c8e458a0a4c55e17150b5e7aa4f not found: ID does not exist" containerID="f3dd63b355e99a4cb2b1c205ecdf56aec2d78c8e458a0a4c55e17150b5e7aa4f" Nov 03 20:48:52.555762 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:52.555728 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3dd63b355e99a4cb2b1c205ecdf56aec2d78c8e458a0a4c55e17150b5e7aa4f"} err="failed to get container status \"f3dd63b355e99a4cb2b1c205ecdf56aec2d78c8e458a0a4c55e17150b5e7aa4f\": rpc error: code = NotFound desc = could not find container \"f3dd63b355e99a4cb2b1c205ecdf56aec2d78c8e458a0a4c55e17150b5e7aa4f\": container with ID starting with f3dd63b355e99a4cb2b1c205ecdf56aec2d78c8e458a0a4c55e17150b5e7aa4f not found: ID does not exist" Nov 03 20:48:52.568849 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:52.568821 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8"] Nov 03 20:48:52.572734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:52.572708 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-e95cf7/simulator-single-main-e95cf7-kserve-5c696d5f7f-qtsl8"] Nov 03 20:48:53.756730 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:53.756696 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11bc51d-2a96-4de7-bfe2-485734b2403b" path="/var/lib/kubelet/pods/d11bc51d-2a96-4de7-bfe2-485734b2403b/volumes" Nov 03 20:48:53.914525 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:53.914491 4457 patch_prober.go:29] interesting pod/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 20:48:53.914717 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:53.914556 4457 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 20:48:54.559209 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.559178 4457 generic.go:356] "Generic (PLEG): container finished" podID="8f41d872-4ee5-45b3-bfab-653323283361" containerID="a8ca7831f44451fd7db8cb60510a9e7f10cb6a9084c706842a30a2316f3a66ed" exitCode=0 Nov 03 20:48:54.559419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.559251 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" event={"ID":"8f41d872-4ee5-45b3-bfab-653323283361","Type":"ContainerDied","Data":"a8ca7831f44451fd7db8cb60510a9e7f10cb6a9084c706842a30a2316f3a66ed"} Nov 03 20:48:54.559419 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.559325 4457 scope.go:122] "RemoveContainer" containerID="b987233571128afcf0ccf1b8b5ce9e5df475f4887d0326bb5cf7465816f19db0" Nov 03 20:48:54.581141 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:48:54.581114 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b987233571128afcf0ccf1b8b5ce9e5df475f4887d0326bb5cf7465816f19db0\": container with ID starting with b987233571128afcf0ccf1b8b5ce9e5df475f4887d0326bb5cf7465816f19db0 not found: ID does not exist" containerID="b987233571128afcf0ccf1b8b5ce9e5df475f4887d0326bb5cf7465816f19db0" Nov 03 20:48:54.584375 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.584357 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:48:54.701898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.701818 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-credential-socket\") pod \"8f41d872-4ee5-45b3-bfab-653323283361\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " Nov 03 20:48:54.701898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.701858 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-kube-api-access-x6szv\" (UniqueName: \"kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-kube-api-access-x6szv\") pod \"8f41d872-4ee5-45b3-bfab-653323283361\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " Nov 03 20:48:54.701898 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.701879 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-certs\") pod \"8f41d872-4ee5-45b3-bfab-653323283361\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " Nov 03 20:48:54.702198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.701915 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-envoy\") pod \"8f41d872-4ee5-45b3-bfab-653323283361\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " Nov 03 20:48:54.702198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.701935 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-istio-token\" (UniqueName: \"kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-istio-token\") pod \"8f41d872-4ee5-45b3-bfab-653323283361\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " Nov 03 20:48:54.702198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.701973 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-socket\") pod \"8f41d872-4ee5-45b3-bfab-653323283361\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " Nov 03 20:48:54.702198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.702009 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-data\") pod \"8f41d872-4ee5-45b3-bfab-653323283361\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " Nov 03 20:48:54.702198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.702073 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f41d872-4ee5-45b3-bfab-653323283361-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f41d872-4ee5-45b3-bfab-653323283361-istiod-ca-cert\") pod \"8f41d872-4ee5-45b3-bfab-653323283361\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " Nov 03 20:48:54.702198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.702152 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-credential-socket" pod "8f41d872-4ee5-45b3-bfab-653323283361" (UID: "8f41d872-4ee5-45b3-bfab-653323283361"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:48:54.702198 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.702159 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f41d872-4ee5-45b3-bfab-653323283361-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f41d872-4ee5-45b3-bfab-653323283361-istio-podinfo\") pod \"8f41d872-4ee5-45b3-bfab-653323283361\" (UID: \"8f41d872-4ee5-45b3-bfab-653323283361\") " Nov 03 20:48:54.702633 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.702345 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-data" pod "8f41d872-4ee5-45b3-bfab-653323283361" (UID: "8f41d872-4ee5-45b3-bfab-653323283361"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:48:54.702633 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.702363 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-socket" pod "8f41d872-4ee5-45b3-bfab-653323283361" (UID: "8f41d872-4ee5-45b3-bfab-653323283361"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:48:54.702633 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.702596 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-certs" pod "8f41d872-4ee5-45b3-bfab-653323283361" (UID: "8f41d872-4ee5-45b3-bfab-653323283361"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:48:54.702633 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.702603 4457 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-credential-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:54.702857 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.702640 4457 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-socket\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:54.702857 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.702655 4457 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-data\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:54.702857 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.702693 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f41d872-4ee5-45b3-bfab-653323283361-istiod-ca-cert" pod "8f41d872-4ee5-45b3-bfab-653323283361" (UID: "8f41d872-4ee5-45b3-bfab-653323283361"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:48:54.704177 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.704148 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f41d872-4ee5-45b3-bfab-653323283361-istio-podinfo" pod "8f41d872-4ee5-45b3-bfab-653323283361" (UID: "8f41d872-4ee5-45b3-bfab-653323283361"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 20:48:54.704312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.704224 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-istio-token" pod "8f41d872-4ee5-45b3-bfab-653323283361" (UID: "8f41d872-4ee5-45b3-bfab-653323283361"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:54.704312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.704238 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-kube-api-access-x6szv" pod "8f41d872-4ee5-45b3-bfab-653323283361" (UID: "8f41d872-4ee5-45b3-bfab-653323283361"). InnerVolumeSpecName "kube-api-access-x6szv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:48:54.704312 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.704281 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-envoy" pod "8f41d872-4ee5-45b3-bfab-653323283361" (UID: "8f41d872-4ee5-45b3-bfab-653323283361"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:48:54.803963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.803841 4457 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f41d872-4ee5-45b3-bfab-653323283361-istio-podinfo\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:54.803963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.803877 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6szv\" (UniqueName: \"kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-kube-api-access-x6szv\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:54.803963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.803897 4457 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-workload-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:54.803963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.803910 4457 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f41d872-4ee5-45b3-bfab-653323283361-istio-envoy\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:54.803963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.803924 4457 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f41d872-4ee5-45b3-bfab-653323283361-istio-token\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:54.803963 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:54.803938 4457 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f41d872-4ee5-45b3-bfab-653323283361-istiod-ca-cert\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:48:55.565520 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:55.565491 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" Nov 03 20:48:55.565708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:55.565534 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt" event={"ID":"8f41d872-4ee5-45b3-bfab-653323283361","Type":"ContainerDied","Data":"ebc15936a0bab785756ae750e565440be8d7fcd09a3d32886c3a6e43c360babf"} Nov 03 20:48:55.565708 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:55.565597 4457 scope.go:122] "RemoveContainer" containerID="a8ca7831f44451fd7db8cb60510a9e7f10cb6a9084c706842a30a2316f3a66ed" Nov 03 20:48:55.588734 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:55.588706 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt"] Nov 03 20:48:55.593138 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:55.593117 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-e95cf7-openshift-default-5c475b5965hzfwt"] Nov 03 20:48:55.755238 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:48:55.755208 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f41d872-4ee5-45b3-bfab-653323283361" path="/var/lib/kubelet/pods/8f41d872-4ee5-45b3-bfab-653323283361/volumes" Nov 03 20:49:06.147983 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:49:06.147940 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:49:12.605046 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:12.605008 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g"] Nov 03 20:49:12.605522 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:12.605202 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" podUID="ee01a4bb-5bbb-4824-820f-12a621202e80" containerName="manager" containerID="cri-o://27c29d6e9d9ed771d1d76c037733b84c81ba674fcb18e36a1a205a126a466dc4" gracePeriod=10 Nov 03 20:49:12.861566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:12.861487 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" Nov 03 20:49:13.026858 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.026815 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee01a4bb-5bbb-4824-820f-12a621202e80-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee01a4bb-5bbb-4824-820f-12a621202e80-extensions-socket-volume\") pod \"ee01a4bb-5bbb-4824-820f-12a621202e80\" (UID: \"ee01a4bb-5bbb-4824-820f-12a621202e80\") " Nov 03 20:49:13.027892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.027169 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee01a4bb-5bbb-4824-820f-12a621202e80-kube-api-access-xtcvc\" (UniqueName: \"kubernetes.io/projected/ee01a4bb-5bbb-4824-820f-12a621202e80-kube-api-access-xtcvc\") pod \"ee01a4bb-5bbb-4824-820f-12a621202e80\" (UID: \"ee01a4bb-5bbb-4824-820f-12a621202e80\") " Nov 03 20:49:13.027892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.027319 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee01a4bb-5bbb-4824-820f-12a621202e80-extensions-socket-volume" pod "ee01a4bb-5bbb-4824-820f-12a621202e80" (UID: "ee01a4bb-5bbb-4824-820f-12a621202e80"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:49:13.027892 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.027697 4457 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee01a4bb-5bbb-4824-820f-12a621202e80-extensions-socket-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:13.036629 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.034022 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee01a4bb-5bbb-4824-820f-12a621202e80-kube-api-access-xtcvc" pod "ee01a4bb-5bbb-4824-820f-12a621202e80" (UID: "ee01a4bb-5bbb-4824-820f-12a621202e80"). InnerVolumeSpecName "kube-api-access-xtcvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:49:13.128636 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.128556 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtcvc\" (UniqueName: \"kubernetes.io/projected/ee01a4bb-5bbb-4824-820f-12a621202e80-kube-api-access-xtcvc\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:49:13.675050 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.675010 4457 generic.go:356] "Generic (PLEG): container finished" podID="ee01a4bb-5bbb-4824-820f-12a621202e80" containerID="27c29d6e9d9ed771d1d76c037733b84c81ba674fcb18e36a1a205a126a466dc4" exitCode=0 Nov 03 20:49:13.675511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.675061 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" Nov 03 20:49:13.675511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.675058 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" event={"ID":"ee01a4bb-5bbb-4824-820f-12a621202e80","Type":"ContainerDied","Data":"27c29d6e9d9ed771d1d76c037733b84c81ba674fcb18e36a1a205a126a466dc4"} Nov 03 20:49:13.675511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.675186 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g" event={"ID":"ee01a4bb-5bbb-4824-820f-12a621202e80","Type":"ContainerDied","Data":"737b65778678b2888355ca4211a2d0efb5709c5bc4b2ed1da15a09bd196c02cc"} Nov 03 20:49:13.675511 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.675216 4457 scope.go:122] "RemoveContainer" containerID="27c29d6e9d9ed771d1d76c037733b84c81ba674fcb18e36a1a205a126a466dc4" Nov 03 20:49:13.687195 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.687170 4457 scope.go:122] "RemoveContainer" containerID="27c29d6e9d9ed771d1d76c037733b84c81ba674fcb18e36a1a205a126a466dc4" Nov 03 20:49:13.687527 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:49:13.687498 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27c29d6e9d9ed771d1d76c037733b84c81ba674fcb18e36a1a205a126a466dc4\": container with ID starting with 27c29d6e9d9ed771d1d76c037733b84c81ba674fcb18e36a1a205a126a466dc4 not found: ID does not exist" containerID="27c29d6e9d9ed771d1d76c037733b84c81ba674fcb18e36a1a205a126a466dc4" Nov 03 20:49:13.687604 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.687540 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27c29d6e9d9ed771d1d76c037733b84c81ba674fcb18e36a1a205a126a466dc4"} err="failed to get container status \"27c29d6e9d9ed771d1d76c037733b84c81ba674fcb18e36a1a205a126a466dc4\": rpc error: code = NotFound desc = could not find container \"27c29d6e9d9ed771d1d76c037733b84c81ba674fcb18e36a1a205a126a466dc4\": container with ID starting with 27c29d6e9d9ed771d1d76c037733b84c81ba674fcb18e36a1a205a126a466dc4 not found: ID does not exist" Nov 03 20:49:13.700166 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.700137 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g"] Nov 03 20:49:13.705495 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.705469 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-8ls6g"] Nov 03 20:49:13.755210 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:49:13.755180 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee01a4bb-5bbb-4824-820f-12a621202e80" path="/var/lib/kubelet/pods/ee01a4bb-5bbb-4824-820f-12a621202e80/volumes" Nov 03 20:49:36.144932 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:49:36.144887 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:50:06.144665 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:50:06.144626 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:50:13.615216 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615174 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh"] Nov 03 20:50:13.615645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615565 4457 cpu_manager.go:414] "containerMap: removing container" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" Nov 03 20:50:13.615645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615577 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" Nov 03 20:50:13.615645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615598 4457 cpu_manager.go:414] "containerMap: removing container" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" Nov 03 20:50:13.615645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615603 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" Nov 03 20:50:13.615645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615611 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d11bc51d-2a96-4de7-bfe2-485734b2403b" containerName="main" Nov 03 20:50:13.615645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615616 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11bc51d-2a96-4de7-bfe2-485734b2403b" containerName="main" Nov 03 20:50:13.615645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615625 4457 cpu_manager.go:414] "containerMap: removing container" podUID="ee01a4bb-5bbb-4824-820f-12a621202e80" containerName="manager" Nov 03 20:50:13.615645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615630 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee01a4bb-5bbb-4824-820f-12a621202e80" containerName="manager" Nov 03 20:50:13.615645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615639 4457 cpu_manager.go:414] "containerMap: removing container" podUID="d561a9c5-1af9-447b-826b-5d0d0785404f" containerName="revoke-keys" Nov 03 20:50:13.615645 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615643 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d561a9c5-1af9-447b-826b-5d0d0785404f" containerName="revoke-keys" Nov 03 20:50:13.616004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615695 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" Nov 03 20:50:13.616004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615704 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f41d872-4ee5-45b3-bfab-653323283361" containerName="istio-proxy" Nov 03 20:50:13.616004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615712 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11bc51d-2a96-4de7-bfe2-485734b2403b" containerName="main" Nov 03 20:50:13.616004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615718 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee01a4bb-5bbb-4824-820f-12a621202e80" containerName="manager" Nov 03 20:50:13.616004 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.615725 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="d561a9c5-1af9-447b-826b-5d0d0785404f" containerName="revoke-keys" Nov 03 20:50:13.619911 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.619893 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh" Nov 03 20:50:13.624713 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.624689 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-kkz74\"" Nov 03 20:50:13.638299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.637247 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh"] Nov 03 20:50:13.742603 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.742558 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/392af69d-c63a-4927-980c-5664b3321224-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6wqqh\" (UID: \"392af69d-c63a-4927-980c-5664b3321224\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh" Nov 03 20:50:13.742792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.742698 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4bvl\" (UniqueName: \"kubernetes.io/projected/392af69d-c63a-4927-980c-5664b3321224-kube-api-access-x4bvl\") pod \"kuadrant-operator-controller-manager-678d87766f-6wqqh\" (UID: \"392af69d-c63a-4927-980c-5664b3321224\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh" Nov 03 20:50:13.843864 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.843818 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4bvl\" (UniqueName: \"kubernetes.io/projected/392af69d-c63a-4927-980c-5664b3321224-kube-api-access-x4bvl\") pod \"kuadrant-operator-controller-manager-678d87766f-6wqqh\" (UID: \"392af69d-c63a-4927-980c-5664b3321224\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh" Nov 03 20:50:13.844049 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.843924 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/392af69d-c63a-4927-980c-5664b3321224-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6wqqh\" (UID: \"392af69d-c63a-4927-980c-5664b3321224\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh" Nov 03 20:50:13.844338 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.844317 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/392af69d-c63a-4927-980c-5664b3321224-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6wqqh\" (UID: \"392af69d-c63a-4927-980c-5664b3321224\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh" Nov 03 20:50:13.856581 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.856550 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4bvl\" (UniqueName: \"kubernetes.io/projected/392af69d-c63a-4927-980c-5664b3321224-kube-api-access-x4bvl\") pod \"kuadrant-operator-controller-manager-678d87766f-6wqqh\" (UID: \"392af69d-c63a-4927-980c-5664b3321224\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh" Nov 03 20:50:13.942817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:13.942746 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh" Nov 03 20:50:14.238768 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:14.238734 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh"] Nov 03 20:50:14.243737 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:50:14.243707 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392af69d_c63a_4927_980c_5664b3321224.slice/crio-fd0529aa3e95a6efa24ed81435ffda2d4a488031209b92f76bb3f30b95d79fc2 WatchSource:0}: Error finding container fd0529aa3e95a6efa24ed81435ffda2d4a488031209b92f76bb3f30b95d79fc2: Status 404 returned error can't find the container with id fd0529aa3e95a6efa24ed81435ffda2d4a488031209b92f76bb3f30b95d79fc2 Nov 03 20:50:14.982697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:14.982596 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh" event={"ID":"392af69d-c63a-4927-980c-5664b3321224","Type":"ContainerStarted","Data":"19d9661d94215d12184759eb333717be5c063b1946a40c93d67db4007bf97745"} Nov 03 20:50:14.982697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:14.982659 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh" Nov 03 20:50:14.982697 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:14.982676 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh" event={"ID":"392af69d-c63a-4927-980c-5664b3321224","Type":"ContainerStarted","Data":"fd0529aa3e95a6efa24ed81435ffda2d4a488031209b92f76bb3f30b95d79fc2"} Nov 03 20:50:15.009114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:15.009065 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh" podStartSLOduration=2.009053192 podStartE2EDuration="2.009053192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:50:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:50:15.006006852 +0000 UTC m=+3193.381603600" watchObservedRunningTime="2026-11-03 20:50:15.009053192 +0000 UTC m=+3193.384649938" Nov 03 20:50:25.990533 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:50:25.990495 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6wqqh" Nov 03 20:50:36.144517 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:50:36.144477 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:51:06.144003 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:51:06.143967 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:51:36.145490 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:51:36.145451 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:51:44.617607 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.617572 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx"] Nov 03 20:51:44.621199 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.621181 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" Nov 03 20:51:44.629353 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.629325 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-466ab7\"" Nov 03 20:51:44.634415 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.634359 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx"] Nov 03 20:51:44.654644 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.654611 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-14db2f29-dockercfg-tbh8p\"" Nov 03 20:51:44.656881 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.656853 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc"] Nov 03 20:51:44.660215 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.660195 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" Nov 03 20:51:44.670224 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.670186 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc"] Nov 03 20:51:44.689812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.689774 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8lp8\" (UniqueName: \"kubernetes.io/projected/dc94519f-b326-4597-a274-25c4b798d4e1-kube-api-access-n8lp8\") pod \"payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc\" (UID: \"dc94519f-b326-4597-a274-25c4b798d4e1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" Nov 03 20:51:44.689926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.689823 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc94519f-b326-4597-a274-25c4b798d4e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc\" (UID: \"dc94519f-b326-4597-a274-25c4b798d4e1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" Nov 03 20:51:44.689987 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.689965 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx\" (UID: \"e6ef06c9-1d3c-41d8-b90f-05702df5a20d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" Nov 03 20:51:44.690031 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.689998 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl6dl\" (UniqueName: \"kubernetes.io/projected/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-kube-api-access-zl6dl\") pod \"payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx\" (UID: \"e6ef06c9-1d3c-41d8-b90f-05702df5a20d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" Nov 03 20:51:44.792765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.791625 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8lp8\" (UniqueName: \"kubernetes.io/projected/dc94519f-b326-4597-a274-25c4b798d4e1-kube-api-access-n8lp8\") pod \"payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc\" (UID: \"dc94519f-b326-4597-a274-25c4b798d4e1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" Nov 03 20:51:44.792765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.791703 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc94519f-b326-4597-a274-25c4b798d4e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc\" (UID: \"dc94519f-b326-4597-a274-25c4b798d4e1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" Nov 03 20:51:44.792765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.791832 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx\" (UID: \"e6ef06c9-1d3c-41d8-b90f-05702df5a20d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" Nov 03 20:51:44.792765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.791910 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl6dl\" (UniqueName: \"kubernetes.io/projected/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-kube-api-access-zl6dl\") pod \"payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx\" (UID: \"e6ef06c9-1d3c-41d8-b90f-05702df5a20d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" Nov 03 20:51:44.793114 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.793066 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc94519f-b326-4597-a274-25c4b798d4e1-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc\" (UID: \"dc94519f-b326-4597-a274-25c4b798d4e1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" Nov 03 20:51:44.796423 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.793843 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx\" (UID: \"e6ef06c9-1d3c-41d8-b90f-05702df5a20d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" Nov 03 20:51:44.804169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.804134 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl6dl\" (UniqueName: \"kubernetes.io/projected/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-kube-api-access-zl6dl\") pod \"payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx\" (UID: \"e6ef06c9-1d3c-41d8-b90f-05702df5a20d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" Nov 03 20:51:44.805036 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.805014 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8lp8\" (UniqueName: \"kubernetes.io/projected/dc94519f-b326-4597-a274-25c4b798d4e1-kube-api-access-n8lp8\") pod \"payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc\" (UID: \"dc94519f-b326-4597-a274-25c4b798d4e1\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" Nov 03 20:51:44.937231 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.937129 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" Nov 03 20:51:44.974472 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:44.974431 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" Nov 03 20:51:45.231254 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:45.231207 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx"] Nov 03 20:51:45.233695 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:51:45.233661 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ef06c9_1d3c_41d8_b90f_05702df5a20d.slice/crio-ee774f63449626b78ea10f5f119aa5799edb2ee8ecd49c8ce3838e8bc2238371 WatchSource:0}: Error finding container ee774f63449626b78ea10f5f119aa5799edb2ee8ecd49c8ce3838e8bc2238371: Status 404 returned error can't find the container with id ee774f63449626b78ea10f5f119aa5799edb2ee8ecd49c8ce3838e8bc2238371 Nov 03 20:51:45.284352 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:45.284327 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc"] Nov 03 20:51:45.286424 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:51:45.286396 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc94519f_b326_4597_a274_25c4b798d4e1.slice/crio-af4190b71e0e2ff7ccbfabbb9b050f9b53ebb1159953412fb472d901c4a66571 WatchSource:0}: Error finding container af4190b71e0e2ff7ccbfabbb9b050f9b53ebb1159953412fb472d901c4a66571: Status 404 returned error can't find the container with id af4190b71e0e2ff7ccbfabbb9b050f9b53ebb1159953412fb472d901c4a66571 Nov 03 20:51:45.501500 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:45.501399 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" event={"ID":"dc94519f-b326-4597-a274-25c4b798d4e1","Type":"ContainerStarted","Data":"7958268114eb4646db21e69c956c56b807a81ac63c88077c2d7020343b1c8ee7"} Nov 03 20:51:45.501500 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:45.501465 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" event={"ID":"dc94519f-b326-4597-a274-25c4b798d4e1","Type":"ContainerStarted","Data":"af4190b71e0e2ff7ccbfabbb9b050f9b53ebb1159953412fb472d901c4a66571"} Nov 03 20:51:45.501746 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:45.501578 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" Nov 03 20:51:45.502956 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:45.502927 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" event={"ID":"e6ef06c9-1d3c-41d8-b90f-05702df5a20d","Type":"ContainerStarted","Data":"29512f65244855ae62da7480f4909e7b975a0a3d08ef49d43b9d1112b63373a6"} Nov 03 20:51:45.503096 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:45.502968 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" event={"ID":"e6ef06c9-1d3c-41d8-b90f-05702df5a20d","Type":"ContainerStarted","Data":"ee774f63449626b78ea10f5f119aa5799edb2ee8ecd49c8ce3838e8bc2238371"} Nov 03 20:51:45.503096 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:45.502989 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" Nov 03 20:51:45.520425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:45.520363 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" podStartSLOduration=1.520350671 podStartE2EDuration="1.520350671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:45.519195239 +0000 UTC m=+3283.894791987" watchObservedRunningTime="2026-11-03 20:51:45.520350671 +0000 UTC m=+3283.895947419" Nov 03 20:51:45.539438 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:45.539373 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" podStartSLOduration=1.539359465 podStartE2EDuration="1.539359465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:45.538524226 +0000 UTC m=+3283.914120974" watchObservedRunningTime="2026-11-03 20:51:45.539359465 +0000 UTC m=+3283.914956211" Nov 03 20:51:55.770849 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:55.770808 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8"] Nov 03 20:51:55.775080 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:55.775052 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:55.782803 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:55.782767 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-466ab7\"/\"kube-root-ca.crt\"" Nov 03 20:51:55.783635 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:55.783608 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-466ab7\"/\"openshift-service-ca.crt\"" Nov 03 20:51:55.785274 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:55.785239 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8"] Nov 03 20:51:55.826871 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:55.826839 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-466ab7\"/\"default-dockercfg-xfsgf\"" Nov 03 20:51:55.826871 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:55.826839 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-466ab7\"/\"simulator-single-main-466ab7-kserve-self-signed-certs\"" Nov 03 20:51:55.905373 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:55.905340 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-tmp-dir\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:55.905591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:55.905382 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-dshm\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:55.905591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:55.905540 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54de1a8a-594f-4f19-94a2-944b275b01ee-tls-certs\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:55.905591 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:55.905570 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-model-cache\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:55.905732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:55.905621 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgcrh\" (UniqueName: \"kubernetes.io/projected/54de1a8a-594f-4f19-94a2-944b275b01ee-kube-api-access-xgcrh\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:55.905732 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:55.905681 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-home\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.006817 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.006779 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgcrh\" (UniqueName: \"kubernetes.io/projected/54de1a8a-594f-4f19-94a2-944b275b01ee-kube-api-access-xgcrh\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.007039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.006832 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-home\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.007039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.006885 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-tmp-dir\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.007039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.006925 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-dshm\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.007039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.006979 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54de1a8a-594f-4f19-94a2-944b275b01ee-tls-certs\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.007039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.007004 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-model-cache\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.007379 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.007352 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-home\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.007463 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.007426 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-tmp-dir\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.007532 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.007477 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-model-cache\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.009313 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.009291 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-dshm\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.009574 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.009555 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54de1a8a-594f-4f19-94a2-944b275b01ee-tls-certs\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.020086 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.020063 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgcrh\" (UniqueName: \"kubernetes.io/projected/54de1a8a-594f-4f19-94a2-944b275b01ee-kube-api-access-xgcrh\") pod \"simulator-single-main-466ab7-kserve-744f9b9954-66cz8\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.098835 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.098793 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:51:56.396791 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.396762 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8"] Nov 03 20:51:56.399045 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:51:56.399020 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54de1a8a_594f_4f19_94a2_944b275b01ee.slice/crio-72effa6fd0773cf9bf71e63f577105134266ffbe266e842fe143d635b8413039 WatchSource:0}: Error finding container 72effa6fd0773cf9bf71e63f577105134266ffbe266e842fe143d635b8413039: Status 404 returned error can't find the container with id 72effa6fd0773cf9bf71e63f577105134266ffbe266e842fe143d635b8413039 Nov 03 20:51:56.512368 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.512340 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" Nov 03 20:51:56.512557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.512443 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" Nov 03 20:51:56.597439 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.597396 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" event={"ID":"54de1a8a-594f-4f19-94a2-944b275b01ee","Type":"ContainerStarted","Data":"2995d12a018d348d7812e9b2def8d55347fad8a69c271213d34c262bcfccf1db"} Nov 03 20:51:56.597618 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.597457 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" event={"ID":"54de1a8a-594f-4f19-94a2-944b275b01ee","Type":"ContainerStarted","Data":"72effa6fd0773cf9bf71e63f577105134266ffbe266e842fe143d635b8413039"} Nov 03 20:51:56.625246 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:56.625180 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" podStartSLOduration=1.625163463 podStartE2EDuration="1.625163463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:51:56.625062021 +0000 UTC m=+3295.000658778" watchObservedRunningTime="2026-11-03 20:51:56.625163463 +0000 UTC m=+3295.000760211" Nov 03 20:51:57.099893 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:51:57.099854 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:52:02.995751 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:02.995715 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:52:03.003093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:03.003068 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:52:03.010118 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:03.010095 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:52:03.017901 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:03.017879 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:52:06.099862 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:06.099825 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:52:06.112287 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:06.112250 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:52:06.145582 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:52:06.145549 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:52:06.669935 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:06.669898 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:52:25.362794 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.362757 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf"] Nov 03 20:52:25.367197 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.367171 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" Nov 03 20:52:25.373070 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.373044 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:52:25.376699 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.373649 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:52:25.379678 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.379472 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf"] Nov 03 20:52:25.393096 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.393060 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb480ea9-4vdcf\" (UID: \"bfd04d54-a855-46f1-a825-2ba3668b0474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" Nov 03 20:52:25.393232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.393189 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h275h\" (UniqueName: \"kubernetes.io/projected/bfd04d54-a855-46f1-a825-2ba3668b0474-kube-api-access-h275h\") pod \"maas-api-revoke-sub-cb480ea9-4vdcf\" (UID: \"bfd04d54-a855-46f1-a825-2ba3668b0474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" Nov 03 20:52:25.393232 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.393216 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb480ea9-4vdcf\" (UID: \"bfd04d54-a855-46f1-a825-2ba3668b0474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" Nov 03 20:52:25.494592 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.494537 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h275h\" (UniqueName: \"kubernetes.io/projected/bfd04d54-a855-46f1-a825-2ba3668b0474-kube-api-access-h275h\") pod \"maas-api-revoke-sub-cb480ea9-4vdcf\" (UID: \"bfd04d54-a855-46f1-a825-2ba3668b0474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" Nov 03 20:52:25.494845 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.494601 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb480ea9-4vdcf\" (UID: \"bfd04d54-a855-46f1-a825-2ba3668b0474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" Nov 03 20:52:25.494845 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.494694 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb480ea9-4vdcf\" (UID: \"bfd04d54-a855-46f1-a825-2ba3668b0474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" Nov 03 20:52:25.495745 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.495714 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb480ea9-4vdcf\" (UID: \"bfd04d54-a855-46f1-a825-2ba3668b0474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" Nov 03 20:52:25.499442 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.499416 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb480ea9-4vdcf\" (UID: \"bfd04d54-a855-46f1-a825-2ba3668b0474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" Nov 03 20:52:25.507634 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.507603 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h275h\" (UniqueName: \"kubernetes.io/projected/bfd04d54-a855-46f1-a825-2ba3668b0474-kube-api-access-h275h\") pod \"maas-api-revoke-sub-cb480ea9-4vdcf\" (UID: \"bfd04d54-a855-46f1-a825-2ba3668b0474\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" Nov 03 20:52:25.688924 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.688846 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" Nov 03 20:52:25.970113 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:25.970077 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf"] Nov 03 20:52:25.971341 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:52:25.971303 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd04d54_a855_46f1_a825_2ba3668b0474.slice/crio-3a776bbef4fe89427ad534c36b740b099257553e09deb9b49b3d58844504f49d WatchSource:0}: Error finding container 3a776bbef4fe89427ad534c36b740b099257553e09deb9b49b3d58844504f49d: Status 404 returned error can't find the container with id 3a776bbef4fe89427ad534c36b740b099257553e09deb9b49b3d58844504f49d Nov 03 20:52:26.773238 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:26.773204 4457 generic.go:356] "Generic (PLEG): container finished" podID="bfd04d54-a855-46f1-a825-2ba3668b0474" containerID="7ffbb0852d85ad38be29508ac377d072bd10b3ef427dabd476890eb312f65216" exitCode=0 Nov 03 20:52:26.773617 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:26.773286 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" event={"ID":"bfd04d54-a855-46f1-a825-2ba3668b0474","Type":"ContainerDied","Data":"7ffbb0852d85ad38be29508ac377d072bd10b3ef427dabd476890eb312f65216"} Nov 03 20:52:26.773617 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:26.773340 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" event={"ID":"bfd04d54-a855-46f1-a825-2ba3668b0474","Type":"ContainerStarted","Data":"3a776bbef4fe89427ad534c36b740b099257553e09deb9b49b3d58844504f49d"} Nov 03 20:52:27.914792 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:27.914765 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" Nov 03 20:52:28.020808 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:28.020773 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-xks-ca\") pod \"bfd04d54-a855-46f1-a825-2ba3668b0474\" (UID: \"bfd04d54-a855-46f1-a825-2ba3668b0474\") " Nov 03 20:52:28.021356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:28.020815 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-openshift-ca\") pod \"bfd04d54-a855-46f1-a825-2ba3668b0474\" (UID: \"bfd04d54-a855-46f1-a825-2ba3668b0474\") " Nov 03 20:52:28.021356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:28.020928 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfd04d54-a855-46f1-a825-2ba3668b0474-kube-api-access-h275h\" (UniqueName: \"kubernetes.io/projected/bfd04d54-a855-46f1-a825-2ba3668b0474-kube-api-access-h275h\") pod \"bfd04d54-a855-46f1-a825-2ba3668b0474\" (UID: \"bfd04d54-a855-46f1-a825-2ba3668b0474\") " Nov 03 20:52:28.021356 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:28.021259 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-openshift-ca" pod "bfd04d54-a855-46f1-a825-2ba3668b0474" (UID: "bfd04d54-a855-46f1-a825-2ba3668b0474"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:28.023052 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:28.023024 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-xks-ca" pod "bfd04d54-a855-46f1-a825-2ba3668b0474" (UID: "bfd04d54-a855-46f1-a825-2ba3668b0474"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:28.023157 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:28.023120 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfd04d54-a855-46f1-a825-2ba3668b0474-kube-api-access-h275h" pod "bfd04d54-a855-46f1-a825-2ba3668b0474" (UID: "bfd04d54-a855-46f1-a825-2ba3668b0474"). InnerVolumeSpecName "kube-api-access-h275h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:28.121843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:28.121806 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h275h\" (UniqueName: \"kubernetes.io/projected/bfd04d54-a855-46f1-a825-2ba3668b0474-kube-api-access-h275h\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:28.121843 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:28.121840 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:28.122054 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:28.121854 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfd04d54-a855-46f1-a825-2ba3668b0474-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:28.784536 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:28.784503 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" Nov 03 20:52:28.784720 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:28.784502 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf" event={"ID":"bfd04d54-a855-46f1-a825-2ba3668b0474","Type":"ContainerDied","Data":"3a776bbef4fe89427ad534c36b740b099257553e09deb9b49b3d58844504f49d"} Nov 03 20:52:28.784720 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:28.784641 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a776bbef4fe89427ad534c36b740b099257553e09deb9b49b3d58844504f49d" Nov 03 20:52:36.149084 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:52:36.148016 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:52:36.321099 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.321064 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6"] Nov 03 20:52:36.321498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.321484 4457 cpu_manager.go:414] "containerMap: removing container" podUID="bfd04d54-a855-46f1-a825-2ba3668b0474" containerName="revoke-keys" Nov 03 20:52:36.321548 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.321499 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfd04d54-a855-46f1-a825-2ba3668b0474" containerName="revoke-keys" Nov 03 20:52:36.321601 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.321592 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfd04d54-a855-46f1-a825-2ba3668b0474" containerName="revoke-keys" Nov 03 20:52:36.324593 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.324577 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" Nov 03 20:52:36.330936 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.330897 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xzp5p\"" Nov 03 20:52:36.332261 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.332234 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 20:52:36.334860 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.334836 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6"] Nov 03 20:52:36.399933 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.399851 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmz9n\" (UniqueName: \"kubernetes.io/projected/b1585e7a-471f-4e36-a44d-7f934ce60944-kube-api-access-zmz9n\") pod \"maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6\" (UID: \"b1585e7a-471f-4e36-a44d-7f934ce60944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" Nov 03 20:52:36.399933 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.399897 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6\" (UID: \"b1585e7a-471f-4e36-a44d-7f934ce60944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" Nov 03 20:52:36.399933 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.399918 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6\" (UID: \"b1585e7a-471f-4e36-a44d-7f934ce60944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" Nov 03 20:52:36.500738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.500698 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmz9n\" (UniqueName: \"kubernetes.io/projected/b1585e7a-471f-4e36-a44d-7f934ce60944-kube-api-access-zmz9n\") pod \"maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6\" (UID: \"b1585e7a-471f-4e36-a44d-7f934ce60944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" Nov 03 20:52:36.500936 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.500747 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6\" (UID: \"b1585e7a-471f-4e36-a44d-7f934ce60944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" Nov 03 20:52:36.500936 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.500768 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6\" (UID: \"b1585e7a-471f-4e36-a44d-7f934ce60944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" Nov 03 20:52:36.501488 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.501455 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6\" (UID: \"b1585e7a-471f-4e36-a44d-7f934ce60944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" Nov 03 20:52:36.503169 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.503145 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6\" (UID: \"b1585e7a-471f-4e36-a44d-7f934ce60944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" Nov 03 20:52:36.512959 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.512930 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmz9n\" (UniqueName: \"kubernetes.io/projected/b1585e7a-471f-4e36-a44d-7f934ce60944-kube-api-access-zmz9n\") pod \"maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6\" (UID: \"b1585e7a-471f-4e36-a44d-7f934ce60944\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" Nov 03 20:52:36.638896 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.638853 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" Nov 03 20:52:36.924596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:36.924568 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6"] Nov 03 20:52:36.926531 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:52:36.926501 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1585e7a_471f_4e36_a44d_7f934ce60944.slice/crio-4cd3cfed9cd010813daaa80ecc8dbd54c3affe1b1ae932a17fca0008fae6d6be WatchSource:0}: Error finding container 4cd3cfed9cd010813daaa80ecc8dbd54c3affe1b1ae932a17fca0008fae6d6be: Status 404 returned error can't find the container with id 4cd3cfed9cd010813daaa80ecc8dbd54c3affe1b1ae932a17fca0008fae6d6be Nov 03 20:52:37.838696 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:37.838661 4457 generic.go:356] "Generic (PLEG): container finished" podID="b1585e7a-471f-4e36-a44d-7f934ce60944" containerID="bc28fd22894f6a5c0f2e81e0da5488c687cf431f96651b4aa4bd7a64aec852ad" exitCode=0 Nov 03 20:52:37.839093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:37.838728 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" event={"ID":"b1585e7a-471f-4e36-a44d-7f934ce60944","Type":"ContainerDied","Data":"bc28fd22894f6a5c0f2e81e0da5488c687cf431f96651b4aa4bd7a64aec852ad"} Nov 03 20:52:37.839093 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:37.838764 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" event={"ID":"b1585e7a-471f-4e36-a44d-7f934ce60944","Type":"ContainerStarted","Data":"4cd3cfed9cd010813daaa80ecc8dbd54c3affe1b1ae932a17fca0008fae6d6be"} Nov 03 20:52:38.977795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:38.977767 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" Nov 03 20:52:39.026314 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:39.026280 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1585e7a-471f-4e36-a44d-7f934ce60944-kube-api-access-zmz9n\" (UniqueName: \"kubernetes.io/projected/b1585e7a-471f-4e36-a44d-7f934ce60944-kube-api-access-zmz9n\") pod \"b1585e7a-471f-4e36-a44d-7f934ce60944\" (UID: \"b1585e7a-471f-4e36-a44d-7f934ce60944\") " Nov 03 20:52:39.026535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:39.026380 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-openshift-ca\") pod \"b1585e7a-471f-4e36-a44d-7f934ce60944\" (UID: \"b1585e7a-471f-4e36-a44d-7f934ce60944\") " Nov 03 20:52:39.026535 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:39.026433 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-xks-ca\") pod \"b1585e7a-471f-4e36-a44d-7f934ce60944\" (UID: \"b1585e7a-471f-4e36-a44d-7f934ce60944\") " Nov 03 20:52:39.026751 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:39.026724 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-openshift-ca" pod "b1585e7a-471f-4e36-a44d-7f934ce60944" (UID: "b1585e7a-471f-4e36-a44d-7f934ce60944"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:39.026901 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:39.026824 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-openshift-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:39.028554 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:39.028529 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-xks-ca" pod "b1585e7a-471f-4e36-a44d-7f934ce60944" (UID: "b1585e7a-471f-4e36-a44d-7f934ce60944"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:39.028964 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:39.028938 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1585e7a-471f-4e36-a44d-7f934ce60944-kube-api-access-zmz9n" pod "b1585e7a-471f-4e36-a44d-7f934ce60944" (UID: "b1585e7a-471f-4e36-a44d-7f934ce60944"). InnerVolumeSpecName "kube-api-access-zmz9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:39.127950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:39.127859 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmz9n\" (UniqueName: \"kubernetes.io/projected/b1585e7a-471f-4e36-a44d-7f934ce60944-kube-api-access-zmz9n\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:39.127950 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:39.127901 4457 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1585e7a-471f-4e36-a44d-7f934ce60944-maas-api-xks-ca\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:39.851452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:39.851427 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" Nov 03 20:52:39.851452 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:39.851435 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6" event={"ID":"b1585e7a-471f-4e36-a44d-7f934ce60944","Type":"ContainerDied","Data":"4cd3cfed9cd010813daaa80ecc8dbd54c3affe1b1ae932a17fca0008fae6d6be"} Nov 03 20:52:39.851675 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:39.851481 4457 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cd3cfed9cd010813daaa80ecc8dbd54c3affe1b1ae932a17fca0008fae6d6be" Nov 03 20:52:41.128739 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:41.128702 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8"] Nov 03 20:52:46.453812 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:46.453769 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc"] Nov 03 20:52:46.471958 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:46.471921 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx"] Nov 03 20:52:47.764200 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:47.764162 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6"] Nov 03 20:52:47.767273 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:47.767249 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-466ab7-wtdm6"] Nov 03 20:52:49.756556 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:49.756519 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1585e7a-471f-4e36-a44d-7f934ce60944" path="/var/lib/kubelet/pods/b1585e7a-471f-4e36-a44d-7f934ce60944/volumes" Nov 03 20:52:51.454649 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.454594 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" podUID="dc94519f-b326-4597-a274-25c4b798d4e1" containerName="payload-processing" containerID="cri-o://7958268114eb4646db21e69c956c56b807a81ac63c88077c2d7020343b1c8ee7" gracePeriod=25 Nov 03 20:52:51.473126 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.473089 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" podUID="e6ef06c9-1d3c-41d8-b90f-05702df5a20d" containerName="payload-pre-processing" containerID="cri-o://29512f65244855ae62da7480f4909e7b975a0a3d08ef49d43b9d1112b63373a6" gracePeriod=25 Nov 03 20:52:51.735798 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.735775 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" Nov 03 20:52:51.746816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.746794 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" Nov 03 20:52:51.853336 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.853299 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-plugins-config-volume\") pod \"e6ef06c9-1d3c-41d8-b90f-05702df5a20d\" (UID: \"e6ef06c9-1d3c-41d8-b90f-05702df5a20d\") " Nov 03 20:52:51.853557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.853383 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc94519f-b326-4597-a274-25c4b798d4e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc94519f-b326-4597-a274-25c4b798d4e1-plugins-config-volume\") pod \"dc94519f-b326-4597-a274-25c4b798d4e1\" (UID: \"dc94519f-b326-4597-a274-25c4b798d4e1\") " Nov 03 20:52:51.853557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.853438 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc94519f-b326-4597-a274-25c4b798d4e1-kube-api-access-n8lp8\" (UniqueName: \"kubernetes.io/projected/dc94519f-b326-4597-a274-25c4b798d4e1-kube-api-access-n8lp8\") pod \"dc94519f-b326-4597-a274-25c4b798d4e1\" (UID: \"dc94519f-b326-4597-a274-25c4b798d4e1\") " Nov 03 20:52:51.853557 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.853513 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-kube-api-access-zl6dl\" (UniqueName: \"kubernetes.io/projected/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-kube-api-access-zl6dl\") pod \"e6ef06c9-1d3c-41d8-b90f-05702df5a20d\" (UID: \"e6ef06c9-1d3c-41d8-b90f-05702df5a20d\") " Nov 03 20:52:51.853789 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.853762 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc94519f-b326-4597-a274-25c4b798d4e1-plugins-config-volume" pod "dc94519f-b326-4597-a274-25c4b798d4e1" (UID: "dc94519f-b326-4597-a274-25c4b798d4e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:51.853789 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.853773 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-plugins-config-volume" pod "e6ef06c9-1d3c-41d8-b90f-05702df5a20d" (UID: "e6ef06c9-1d3c-41d8-b90f-05702df5a20d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 20:52:51.855707 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.855670 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc94519f-b326-4597-a274-25c4b798d4e1-kube-api-access-n8lp8" pod "dc94519f-b326-4597-a274-25c4b798d4e1" (UID: "dc94519f-b326-4597-a274-25c4b798d4e1"). InnerVolumeSpecName "kube-api-access-n8lp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:51.855859 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.855837 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-kube-api-access-zl6dl" pod "e6ef06c9-1d3c-41d8-b90f-05702df5a20d" (UID: "e6ef06c9-1d3c-41d8-b90f-05702df5a20d"). InnerVolumeSpecName "kube-api-access-zl6dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:51.921469 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.921435 4457 generic.go:356] "Generic (PLEG): container finished" podID="dc94519f-b326-4597-a274-25c4b798d4e1" containerID="7958268114eb4646db21e69c956c56b807a81ac63c88077c2d7020343b1c8ee7" exitCode=0 Nov 03 20:52:51.921663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.921496 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" Nov 03 20:52:51.921663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.921566 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" event={"ID":"dc94519f-b326-4597-a274-25c4b798d4e1","Type":"ContainerDied","Data":"7958268114eb4646db21e69c956c56b807a81ac63c88077c2d7020343b1c8ee7"} Nov 03 20:52:51.921663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.921628 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc" event={"ID":"dc94519f-b326-4597-a274-25c4b798d4e1","Type":"ContainerDied","Data":"af4190b71e0e2ff7ccbfabbb9b050f9b53ebb1159953412fb472d901c4a66571"} Nov 03 20:52:51.921663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.921661 4457 scope.go:122] "RemoveContainer" containerID="7958268114eb4646db21e69c956c56b807a81ac63c88077c2d7020343b1c8ee7" Nov 03 20:52:51.923175 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.923152 4457 generic.go:356] "Generic (PLEG): container finished" podID="e6ef06c9-1d3c-41d8-b90f-05702df5a20d" containerID="29512f65244855ae62da7480f4909e7b975a0a3d08ef49d43b9d1112b63373a6" exitCode=0 Nov 03 20:52:51.923278 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.923231 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" Nov 03 20:52:51.923345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.923292 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" event={"ID":"e6ef06c9-1d3c-41d8-b90f-05702df5a20d","Type":"ContainerDied","Data":"29512f65244855ae62da7480f4909e7b975a0a3d08ef49d43b9d1112b63373a6"} Nov 03 20:52:51.923345 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.923330 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx" event={"ID":"e6ef06c9-1d3c-41d8-b90f-05702df5a20d","Type":"ContainerDied","Data":"ee774f63449626b78ea10f5f119aa5799edb2ee8ecd49c8ce3838e8bc2238371"} Nov 03 20:52:51.934046 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.934017 4457 scope.go:122] "RemoveContainer" containerID="7958268114eb4646db21e69c956c56b807a81ac63c88077c2d7020343b1c8ee7" Nov 03 20:52:51.934373 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:52:51.934350 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7958268114eb4646db21e69c956c56b807a81ac63c88077c2d7020343b1c8ee7\": container with ID starting with 7958268114eb4646db21e69c956c56b807a81ac63c88077c2d7020343b1c8ee7 not found: ID does not exist" containerID="7958268114eb4646db21e69c956c56b807a81ac63c88077c2d7020343b1c8ee7" Nov 03 20:52:51.934502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.934381 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7958268114eb4646db21e69c956c56b807a81ac63c88077c2d7020343b1c8ee7"} err="failed to get container status \"7958268114eb4646db21e69c956c56b807a81ac63c88077c2d7020343b1c8ee7\": rpc error: code = NotFound desc = could not find container \"7958268114eb4646db21e69c956c56b807a81ac63c88077c2d7020343b1c8ee7\": container with ID starting with 7958268114eb4646db21e69c956c56b807a81ac63c88077c2d7020343b1c8ee7 not found: ID does not exist" Nov 03 20:52:51.934502 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.934421 4457 scope.go:122] "RemoveContainer" containerID="29512f65244855ae62da7480f4909e7b975a0a3d08ef49d43b9d1112b63373a6" Nov 03 20:52:51.943053 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.943036 4457 scope.go:122] "RemoveContainer" containerID="29512f65244855ae62da7480f4909e7b975a0a3d08ef49d43b9d1112b63373a6" Nov 03 20:52:51.943336 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:52:51.943312 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29512f65244855ae62da7480f4909e7b975a0a3d08ef49d43b9d1112b63373a6\": container with ID starting with 29512f65244855ae62da7480f4909e7b975a0a3d08ef49d43b9d1112b63373a6 not found: ID does not exist" containerID="29512f65244855ae62da7480f4909e7b975a0a3d08ef49d43b9d1112b63373a6" Nov 03 20:52:51.943443 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.943340 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29512f65244855ae62da7480f4909e7b975a0a3d08ef49d43b9d1112b63373a6"} err="failed to get container status \"29512f65244855ae62da7480f4909e7b975a0a3d08ef49d43b9d1112b63373a6\": rpc error: code = NotFound desc = could not find container \"29512f65244855ae62da7480f4909e7b975a0a3d08ef49d43b9d1112b63373a6\": container with ID starting with 29512f65244855ae62da7480f4909e7b975a0a3d08ef49d43b9d1112b63373a6 not found: ID does not exist" Nov 03 20:52:51.949214 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.949194 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx"] Nov 03 20:52:51.953441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.953410 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-466ab7-f566c5czqx"] Nov 03 20:52:51.954270 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.954254 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.954330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.954275 4457 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc94519f-b326-4597-a274-25c4b798d4e1-plugins-config-volume\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.954330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.954285 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8lp8\" (UniqueName: \"kubernetes.io/projected/dc94519f-b326-4597-a274-25c4b798d4e1-kube-api-access-n8lp8\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.954330 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.954294 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl6dl\" (UniqueName: \"kubernetes.io/projected/e6ef06c9-1d3c-41d8-b90f-05702df5a20d-kube-api-access-zl6dl\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:51.964744 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.964681 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc"] Nov 03 20:52:51.969221 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:51.969200 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-466ab7-54567db4dv7rvc"] Nov 03 20:52:53.755891 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:53.755858 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc94519f-b326-4597-a274-25c4b798d4e1" path="/var/lib/kubelet/pods/dc94519f-b326-4597-a274-25c4b798d4e1/volumes" Nov 03 20:52:53.756298 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:53.756242 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ef06c9-1d3c-41d8-b90f-05702df5a20d" path="/var/lib/kubelet/pods/e6ef06c9-1d3c-41d8-b90f-05702df5a20d/volumes" Nov 03 20:52:56.148957 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.148883 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" podUID="54de1a8a-594f-4f19-94a2-944b275b01ee" containerName="main" containerID="cri-o://2995d12a018d348d7812e9b2def8d55347fad8a69c271213d34c262bcfccf1db" gracePeriod=45 Nov 03 20:52:56.393851 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.393824 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:52:56.498704 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.498608 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-home\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-home\") pod \"54de1a8a-594f-4f19-94a2-944b275b01ee\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " Nov 03 20:52:56.498886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.498749 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54de1a8a-594f-4f19-94a2-944b275b01ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/54de1a8a-594f-4f19-94a2-944b275b01ee-tls-certs\") pod \"54de1a8a-594f-4f19-94a2-944b275b01ee\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " Nov 03 20:52:56.498886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.498808 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54de1a8a-594f-4f19-94a2-944b275b01ee-kube-api-access-xgcrh\" (UniqueName: \"kubernetes.io/projected/54de1a8a-594f-4f19-94a2-944b275b01ee-kube-api-access-xgcrh\") pod \"54de1a8a-594f-4f19-94a2-944b275b01ee\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " Nov 03 20:52:56.498886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.498839 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-dshm\") pod \"54de1a8a-594f-4f19-94a2-944b275b01ee\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " Nov 03 20:52:56.498886 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.498880 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-tmp-dir\") pod \"54de1a8a-594f-4f19-94a2-944b275b01ee\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " Nov 03 20:52:56.499795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.498890 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-home" pod "54de1a8a-594f-4f19-94a2-944b275b01ee" (UID: "54de1a8a-594f-4f19-94a2-944b275b01ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:56.499795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.498915 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-model-cache\") pod \"54de1a8a-594f-4f19-94a2-944b275b01ee\" (UID: \"54de1a8a-594f-4f19-94a2-944b275b01ee\") " Nov 03 20:52:56.499795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.499179 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-tmp-dir" pod "54de1a8a-594f-4f19-94a2-944b275b01ee" (UID: "54de1a8a-594f-4f19-94a2-944b275b01ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:56.499795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.499239 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:56.499795 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.499325 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-model-cache" pod "54de1a8a-594f-4f19-94a2-944b275b01ee" (UID: "54de1a8a-594f-4f19-94a2-944b275b01ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:56.501441 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.501376 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-dshm" pod "54de1a8a-594f-4f19-94a2-944b275b01ee" (UID: "54de1a8a-594f-4f19-94a2-944b275b01ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:52:56.501566 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.501451 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54de1a8a-594f-4f19-94a2-944b275b01ee-tls-certs" pod "54de1a8a-594f-4f19-94a2-944b275b01ee" (UID: "54de1a8a-594f-4f19-94a2-944b275b01ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:52:56.501626 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.501575 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54de1a8a-594f-4f19-94a2-944b275b01ee-kube-api-access-xgcrh" pod "54de1a8a-594f-4f19-94a2-944b275b01ee" (UID: "54de1a8a-594f-4f19-94a2-944b275b01ee"). InnerVolumeSpecName "kube-api-access-xgcrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:52:56.600066 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.600029 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:56.600066 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.600061 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:56.600066 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.600072 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54de1a8a-594f-4f19-94a2-944b275b01ee-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:56.600307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.600082 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgcrh\" (UniqueName: \"kubernetes.io/projected/54de1a8a-594f-4f19-94a2-944b275b01ee-kube-api-access-xgcrh\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:56.600307 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.600093 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54de1a8a-594f-4f19-94a2-944b275b01ee-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:52:56.954821 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.954788 4457 generic.go:356] "Generic (PLEG): container finished" podID="54de1a8a-594f-4f19-94a2-944b275b01ee" containerID="2995d12a018d348d7812e9b2def8d55347fad8a69c271213d34c262bcfccf1db" exitCode=0 Nov 03 20:52:56.955026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.954831 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" Nov 03 20:52:56.955026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.954830 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" event={"ID":"54de1a8a-594f-4f19-94a2-944b275b01ee","Type":"ContainerDied","Data":"2995d12a018d348d7812e9b2def8d55347fad8a69c271213d34c262bcfccf1db"} Nov 03 20:52:56.955026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.954953 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8" event={"ID":"54de1a8a-594f-4f19-94a2-944b275b01ee","Type":"ContainerDied","Data":"72effa6fd0773cf9bf71e63f577105134266ffbe266e842fe143d635b8413039"} Nov 03 20:52:56.955026 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.954980 4457 scope.go:122] "RemoveContainer" containerID="2995d12a018d348d7812e9b2def8d55347fad8a69c271213d34c262bcfccf1db" Nov 03 20:52:56.967995 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.967973 4457 scope.go:122] "RemoveContainer" containerID="2995d12a018d348d7812e9b2def8d55347fad8a69c271213d34c262bcfccf1db" Nov 03 20:52:56.968473 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:52:56.968442 4457 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2995d12a018d348d7812e9b2def8d55347fad8a69c271213d34c262bcfccf1db\": container with ID starting with 2995d12a018d348d7812e9b2def8d55347fad8a69c271213d34c262bcfccf1db not found: ID does not exist" containerID="2995d12a018d348d7812e9b2def8d55347fad8a69c271213d34c262bcfccf1db" Nov 03 20:52:56.968606 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.968482 4457 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2995d12a018d348d7812e9b2def8d55347fad8a69c271213d34c262bcfccf1db"} err="failed to get container status \"2995d12a018d348d7812e9b2def8d55347fad8a69c271213d34c262bcfccf1db\": rpc error: code = NotFound desc = could not find container \"2995d12a018d348d7812e9b2def8d55347fad8a69c271213d34c262bcfccf1db\": container with ID starting with 2995d12a018d348d7812e9b2def8d55347fad8a69c271213d34c262bcfccf1db not found: ID does not exist" Nov 03 20:52:56.978299 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.978276 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8"] Nov 03 20:52:56.983558 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:56.983536 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-466ab7/simulator-single-main-466ab7-kserve-744f9b9954-66cz8"] Nov 03 20:52:57.756323 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:57.756291 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54de1a8a-594f-4f19-94a2-944b275b01ee" path="/var/lib/kubelet/pods/54de1a8a-594f-4f19-94a2-944b275b01ee/volumes" Nov 03 20:52:58.258848 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.258793 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm"] Nov 03 20:52:58.259440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.259415 4457 cpu_manager.go:414] "containerMap: removing container" podUID="b1585e7a-471f-4e36-a44d-7f934ce60944" containerName="revoke-keys" Nov 03 20:52:58.259440 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.259437 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1585e7a-471f-4e36-a44d-7f934ce60944" containerName="revoke-keys" Nov 03 20:52:58.259627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.259455 4457 cpu_manager.go:414] "containerMap: removing container" podUID="e6ef06c9-1d3c-41d8-b90f-05702df5a20d" containerName="payload-pre-processing" Nov 03 20:52:58.259627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.259463 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ef06c9-1d3c-41d8-b90f-05702df5a20d" containerName="payload-pre-processing" Nov 03 20:52:58.259627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.259475 4457 cpu_manager.go:414] "containerMap: removing container" podUID="54de1a8a-594f-4f19-94a2-944b275b01ee" containerName="main" Nov 03 20:52:58.259627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.259483 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54de1a8a-594f-4f19-94a2-944b275b01ee" containerName="main" Nov 03 20:52:58.259627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.259505 4457 cpu_manager.go:414] "containerMap: removing container" podUID="dc94519f-b326-4597-a274-25c4b798d4e1" containerName="payload-processing" Nov 03 20:52:58.259627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.259512 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc94519f-b326-4597-a274-25c4b798d4e1" containerName="payload-processing" Nov 03 20:52:58.259627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.259592 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1585e7a-471f-4e36-a44d-7f934ce60944" containerName="revoke-keys" Nov 03 20:52:58.259627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.259602 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ef06c9-1d3c-41d8-b90f-05702df5a20d" containerName="payload-pre-processing" Nov 03 20:52:58.259627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.259608 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="54de1a8a-594f-4f19-94a2-944b275b01ee" containerName="main" Nov 03 20:52:58.259627 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.259617 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc94519f-b326-4597-a274-25c4b798d4e1" containerName="payload-processing" Nov 03 20:52:58.264234 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.264202 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.273088 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.273063 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 20:52:58.274230 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.274205 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 20:52:58.274351 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.274269 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm"] Nov 03 20:52:58.280201 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.280178 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-ab04f05f-kserve-self-signed-certs\"" Nov 03 20:52:58.280313 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.280248 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6vbjn\"" Nov 03 20:52:58.419076 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.419033 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-home\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.419479 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.419433 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tls-certs\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.419712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.419523 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-model-cache\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.419712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.419575 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-dshm\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.419712 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.419677 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29jm6\" (UniqueName: \"kubernetes.io/projected/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-kube-api-access-29jm6\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.419900 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.419726 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tmp-dir\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.520336 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.520252 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-home\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.520336 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.520301 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tls-certs\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.520336 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.520318 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-model-cache\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.520641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.520352 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-dshm\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.520641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.520422 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29jm6\" (UniqueName: \"kubernetes.io/projected/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-kube-api-access-29jm6\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.520641 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.520465 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tmp-dir\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.520794 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.520764 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-model-cache\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.520794 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.520783 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-home\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.521095 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.521063 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tmp-dir\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.522737 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.522716 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-dshm\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.523039 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.523023 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tls-certs\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.532680 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.532657 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29jm6\" (UniqueName: \"kubernetes.io/projected/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-kube-api-access-29jm6\") pod \"e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.584656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.584621 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:52:58.862024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.861997 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm"] Nov 03 20:52:58.863944 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:52:58.863914 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb17c0be4_6c99_4df7_bbe3_e36ec6a736b4.slice/crio-702e733c1b6b1748ec30e203825f50c15547d1d27b0df30833713ee8d40fc496 WatchSource:0}: Error finding container 702e733c1b6b1748ec30e203825f50c15547d1d27b0df30833713ee8d40fc496: Status 404 returned error can't find the container with id 702e733c1b6b1748ec30e203825f50c15547d1d27b0df30833713ee8d40fc496 Nov 03 20:52:58.968022 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.967989 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" event={"ID":"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4","Type":"ContainerStarted","Data":"70bc0a6e500e8174d6521feee142d4ac1912bf6ecd40b64e077329c54bdb1c1b"} Nov 03 20:52:58.968209 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.968033 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" event={"ID":"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4","Type":"ContainerStarted","Data":"702e733c1b6b1748ec30e203825f50c15547d1d27b0df30833713ee8d40fc496"} Nov 03 20:52:58.988344 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:58.988290 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" podStartSLOduration=0.988277527 podStartE2EDuration="988.277527ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:52:58.987716975 +0000 UTC m=+3357.363313726" watchObservedRunningTime="2026-11-03 20:52:58.988277527 +0000 UTC m=+3357.363874274" Nov 03 20:52:59.585143 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:52:59.585100 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:53:06.143722 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:53:06.143684 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:53:08.584819 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:08.584778 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:53:08.597251 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:08.597221 4457 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:53:09.058083 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:09.058054 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:53:24.023951 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:24.023863 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j"] Nov 03 20:53:24.026797 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:24.026772 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d90d32e-ws54j"] Nov 03 20:53:25.755766 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:25.755730 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc35d6f0-1471-4daf-9549-ccc179797c93" path="/var/lib/kubelet/pods/bc35d6f0-1471-4daf-9549-ccc179797c93/volumes" Nov 03 20:53:36.279722 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:53:36.279658 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:53:43.223572 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:43.223533 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm"] Nov 03 20:53:58.244365 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.244302 4457 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" podUID="b17c0be4-6c99-4df7-bbe3-e36ec6a736b4" containerName="main" containerID="cri-o://70bc0a6e500e8174d6521feee142d4ac1912bf6ecd40b64e077329c54bdb1c1b" gracePeriod=45 Nov 03 20:53:58.470656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.470170 4457 generic.go:356] "Generic (PLEG): container finished" podID="b17c0be4-6c99-4df7-bbe3-e36ec6a736b4" containerID="70bc0a6e500e8174d6521feee142d4ac1912bf6ecd40b64e077329c54bdb1c1b" exitCode=0 Nov 03 20:53:58.470656 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.470382 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" event={"ID":"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4","Type":"ContainerDied","Data":"70bc0a6e500e8174d6521feee142d4ac1912bf6ecd40b64e077329c54bdb1c1b"} Nov 03 20:53:58.547024 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.546989 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:53:58.624249 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.624217 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-kube-api-access-29jm6\" (UniqueName: \"kubernetes.io/projected/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-kube-api-access-29jm6\") pod \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " Nov 03 20:53:58.624831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.624273 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-model-cache\") pod \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " Nov 03 20:53:58.624831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.624309 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tls-certs\") pod \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " Nov 03 20:53:58.624831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.624383 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-dshm\") pod \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " Nov 03 20:53:58.624831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.624452 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-home\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-home\") pod \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " Nov 03 20:53:58.624831 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.624543 4457 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tmp-dir\") pod \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\" (UID: \"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4\") " Nov 03 20:53:58.625271 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.625231 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-model-cache" pod "b17c0be4-6c99-4df7-bbe3-e36ec6a736b4" (UID: "b17c0be4-6c99-4df7-bbe3-e36ec6a736b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:58.625889 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.625844 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-home" pod "b17c0be4-6c99-4df7-bbe3-e36ec6a736b4" (UID: "b17c0be4-6c99-4df7-bbe3-e36ec6a736b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:58.626302 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.626280 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tmp-dir" pod "b17c0be4-6c99-4df7-bbe3-e36ec6a736b4" (UID: "b17c0be4-6c99-4df7-bbe3-e36ec6a736b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:58.628077 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.628049 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-kube-api-access-29jm6" pod "b17c0be4-6c99-4df7-bbe3-e36ec6a736b4" (UID: "b17c0be4-6c99-4df7-bbe3-e36ec6a736b4"). InnerVolumeSpecName "kube-api-access-29jm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 20:53:58.629806 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.629774 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-dshm" pod "b17c0be4-6c99-4df7-bbe3-e36ec6a736b4" (UID: "b17c0be4-6c99-4df7-bbe3-e36ec6a736b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 20:53:58.629880 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.629789 4457 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tls-certs" pod "b17c0be4-6c99-4df7-bbe3-e36ec6a736b4" (UID: "b17c0be4-6c99-4df7-bbe3-e36ec6a736b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 20:53:58.725514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.725468 4457 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-dshm\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:58.725514 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.725509 4457 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-home\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:58.725816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.725551 4457 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tmp-dir\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:58.725816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.725568 4457 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29jm6\" (UniqueName: \"kubernetes.io/projected/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-kube-api-access-29jm6\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:58.725816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.725585 4457 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-model-cache\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:58.725816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:58.725600 4457 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4-tls-certs\") on node \"ip-10-0-132-25.us-east-2.compute.internal\" DevicePath \"\"" Nov 03 20:53:59.500409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:59.496835 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" event={"ID":"b17c0be4-6c99-4df7-bbe3-e36ec6a736b4","Type":"ContainerDied","Data":"702e733c1b6b1748ec30e203825f50c15547d1d27b0df30833713ee8d40fc496"} Nov 03 20:53:59.500409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:59.496897 4457 scope.go:122] "RemoveContainer" containerID="70bc0a6e500e8174d6521feee142d4ac1912bf6ecd40b64e077329c54bdb1c1b" Nov 03 20:53:59.500409 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:59.497075 4457 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm" Nov 03 20:53:59.559623 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:59.559547 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm"] Nov 03 20:53:59.580418 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:59.579306 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-ab04f05f-kserve-74bf794fb9-cbkrm"] Nov 03 20:53:59.785460 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:53:59.780270 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b17c0be4-6c99-4df7-bbe3-e36ec6a736b4" path="/var/lib/kubelet/pods/b17c0be4-6c99-4df7-bbe3-e36ec6a736b4/volumes" Nov 03 20:54:06.147740 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:54:06.147695 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:54:36.144416 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:54:36.144359 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:54:36.869497 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:36.869461 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw"] Nov 03 20:54:36.869913 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:36.869898 4457 cpu_manager.go:414] "containerMap: removing container" podUID="b17c0be4-6c99-4df7-bbe3-e36ec6a736b4" containerName="main" Nov 03 20:54:36.869986 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:36.869914 4457 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b17c0be4-6c99-4df7-bbe3-e36ec6a736b4" containerName="main" Nov 03 20:54:36.869986 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:36.869979 4457 memory_manager.go:365] "RemoveStaleState removing state" podUID="b17c0be4-6c99-4df7-bbe3-e36ec6a736b4" containerName="main" Nov 03 20:54:36.873100 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:36.873082 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:36.886816 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:36.883981 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw"] Nov 03 20:54:36.902425 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:36.902378 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-znhr8\"" Nov 03 20:54:36.902602 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:36.902439 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 20:54:36.902602 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:36.902483 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 20:54:36.903870 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:36.903850 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2857ffa5-3e45-40b8-a302-86f5083b930c-openshift-service-ca\") pod \"maas-api-78bbd54c6f-qsnrw\" (UID: \"2857ffa5-3e45-40b8-a302-86f5083b930c\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:36.904000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:36.903888 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n6lb\" (UniqueName: \"kubernetes.io/projected/2857ffa5-3e45-40b8-a302-86f5083b930c-kube-api-access-5n6lb\") pod \"maas-api-78bbd54c6f-qsnrw\" (UID: \"2857ffa5-3e45-40b8-a302-86f5083b930c\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:36.904000 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:36.903931 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2857ffa5-3e45-40b8-a302-86f5083b930c-maas-api-tls\") pod \"maas-api-78bbd54c6f-qsnrw\" (UID: \"2857ffa5-3e45-40b8-a302-86f5083b930c\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:36.904115 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:36.904093 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2857ffa5-3e45-40b8-a302-86f5083b930c-metrics-certs\") pod \"maas-api-78bbd54c6f-qsnrw\" (UID: \"2857ffa5-3e45-40b8-a302-86f5083b930c\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:37.005226 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:37.005187 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2857ffa5-3e45-40b8-a302-86f5083b930c-maas-api-tls\") pod \"maas-api-78bbd54c6f-qsnrw\" (UID: \"2857ffa5-3e45-40b8-a302-86f5083b930c\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:37.005468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:37.005249 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2857ffa5-3e45-40b8-a302-86f5083b930c-metrics-certs\") pod \"maas-api-78bbd54c6f-qsnrw\" (UID: \"2857ffa5-3e45-40b8-a302-86f5083b930c\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:37.005468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:37.005294 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2857ffa5-3e45-40b8-a302-86f5083b930c-openshift-service-ca\") pod \"maas-api-78bbd54c6f-qsnrw\" (UID: \"2857ffa5-3e45-40b8-a302-86f5083b930c\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:37.005468 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:37.005340 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n6lb\" (UniqueName: \"kubernetes.io/projected/2857ffa5-3e45-40b8-a302-86f5083b930c-kube-api-access-5n6lb\") pod \"maas-api-78bbd54c6f-qsnrw\" (UID: \"2857ffa5-3e45-40b8-a302-86f5083b930c\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:37.006012 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:37.005989 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2857ffa5-3e45-40b8-a302-86f5083b930c-openshift-service-ca\") pod \"maas-api-78bbd54c6f-qsnrw\" (UID: \"2857ffa5-3e45-40b8-a302-86f5083b930c\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:37.007836 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:37.007811 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2857ffa5-3e45-40b8-a302-86f5083b930c-maas-api-tls\") pod \"maas-api-78bbd54c6f-qsnrw\" (UID: \"2857ffa5-3e45-40b8-a302-86f5083b930c\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:37.007926 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:37.007871 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2857ffa5-3e45-40b8-a302-86f5083b930c-metrics-certs\") pod \"maas-api-78bbd54c6f-qsnrw\" (UID: \"2857ffa5-3e45-40b8-a302-86f5083b930c\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:37.019333 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:37.019313 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n6lb\" (UniqueName: \"kubernetes.io/projected/2857ffa5-3e45-40b8-a302-86f5083b930c-kube-api-access-5n6lb\") pod \"maas-api-78bbd54c6f-qsnrw\" (UID: \"2857ffa5-3e45-40b8-a302-86f5083b930c\") " pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:37.188576 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:37.188485 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:37.471857 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:37.471825 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw"] Nov 03 20:54:37.473085 ip-10-0-132-25 kubenswrapper[4457]: W1103 20:54:37.473051 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2857ffa5_3e45_40b8_a302_86f5083b930c.slice/crio-df90bf702ada4eee4d774cc560a7fc4cb12ec7b4851c60e6e42dc80c2d1b25af WatchSource:0}: Error finding container df90bf702ada4eee4d774cc560a7fc4cb12ec7b4851c60e6e42dc80c2d1b25af: Status 404 returned error can't find the container with id df90bf702ada4eee4d774cc560a7fc4cb12ec7b4851c60e6e42dc80c2d1b25af Nov 03 20:54:37.474524 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:37.474501 4457 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 20:54:37.709220 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:37.709178 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" event={"ID":"2857ffa5-3e45-40b8-a302-86f5083b930c","Type":"ContainerStarted","Data":"df90bf702ada4eee4d774cc560a7fc4cb12ec7b4851c60e6e42dc80c2d1b25af"} Nov 03 20:54:38.719013 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:38.718979 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" event={"ID":"2857ffa5-3e45-40b8-a302-86f5083b930c","Type":"ContainerStarted","Data":"9d6e0c3988b90ae66ce9f98e81cfdd5edd91eb038bf32fba504d3d44852091b0"} Nov 03 20:54:38.719492 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:38.719128 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:54:38.740272 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:38.740224 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" podStartSLOduration=2.326619439 podStartE2EDuration="2.740210866s" totalImagesPullingTime="413.591427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 20:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 20:54:38.739208435 +0000 UTC m=+3457.114805182" watchObservedRunningTime="2026-11-03 20:54:38.740210866 +0000 UTC m=+3457.115807613" Nov 03 20:54:44.730222 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:54:44.730185 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78bbd54c6f-qsnrw" Nov 03 20:55:06.145496 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:55:06.145458 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:55:08.824115 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:55:08.824082 4457 scope.go:122] "RemoveContainer" containerID="78f007bd15c65537d98b509312a35d2cc91295567fe28d766667316ed03b4ff6" Nov 03 20:55:08.833138 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:55:08.833119 4457 scope.go:122] "RemoveContainer" containerID="57fa138312f17c4dc480954246f688793984e0ef27cbb10b6b53852e5c790b61" Nov 03 20:55:36.145408 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:55:36.145346 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:56:06.144076 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:56:06.144036 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:56:36.144247 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:56:36.144209 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:57:03.033058 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:57:03.033028 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:57:03.042691 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:57:03.042657 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-marketplace_c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cbx4mv_2a47a585-8e30-4ec0-b34d-48477530d0ad/extract/0.log" Nov 03 20:57:03.049008 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:57:03.048982 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:57:03.058541 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:57:03.058513 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc9nt_af954863-9eb8-4bd6-a64b-918f5f4ff6b0/ovn-acl-logging/0.log" Nov 03 20:57:06.144429 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:57:06.144376 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:57:28.028917 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:57:28.028880 4457 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf"] Nov 03 20:57:28.030825 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:57:28.030804 4457 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb480ea9-4vdcf"] Nov 03 20:57:29.755289 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:57:29.755251 4457 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfd04d54-a855-46f1-a825-2ba3668b0474" path="/var/lib/kubelet/pods/bfd04d54-a855-46f1-a825-2ba3668b0474/volumes" Nov 03 20:57:36.144643 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:57:36.144592 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:58:06.274802 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:58:06.274742 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:58:16.276498 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:16.276463 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jz4d7_428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb/manager/0.log" Nov 03 20:58:16.749361 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:16.749330 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mq8tg_2e100b28-7be2-4dcf-abbc-3532f79ff178/server/0.log" Nov 03 20:58:17.156633 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:17.156603 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cbnpv_4c6a37ee-a6e8-4345-990b-82ef4d3d3a77/proxy/0.log" Nov 03 20:58:18.119491 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:18.119459 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78bbd54c6f-qsnrw_2857ffa5-3e45-40b8-a302-86f5083b930c/maas-api/0.log" Nov 03 20:58:19.357844 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:19.357810 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-g6q2z_704584c9-f1c3-4968-bffe-92125ac2a96c/postgres/0.log" Nov 03 20:58:21.097028 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:21.096993 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5gww6_a6e7ed4a-ecf6-4a3a-8037-e10cba476618/manager/0.log" Nov 03 20:58:21.675096 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:21.675061 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6wqqh_392af69d-c63a-4927-980c-5664b3321224/manager/0.log" Nov 03 20:58:22.066663 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:22.066618 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-jth7z_603fa1ee-3000-485f-92ad-66a3f1bd726c/manager/0.log" Nov 03 20:58:22.853602 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:22.853570 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-qrjl6_82519fb3-ab4a-42bf-a8e0-c911224c33ab/istio-proxy/0.log" Nov 03 20:58:23.812809 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:23.812771 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-dgjgp_8f7c5959-df8b-4348-8c8c-c61ba06f66d4/istio-proxy/0.log" Nov 03 20:58:24.002765 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:24.002678 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-g6znk_ad84ee89-c964-4229-be80-bb309afb1255/istio-proxy/0.log" Nov 03 20:58:24.389528 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:24.389492 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-n42qd_2e3b005c-f816-4fbd-8812-b007b7fce9e7/istio-proxy/0.log" Nov 03 20:58:24.759690 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:24.759597 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-p7lrd_93f7f2c5-96a8-4ccc-ad9b-3cc648f85802/istio-proxy/1.log" Nov 03 20:58:25.489889 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:25.489856 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59_80d281fa-6f07-44cf-a71f-2de479e23415/istio-proxy/0.log" Nov 03 20:58:26.497744 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:26.497706 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-p7lrd_93f7f2c5-96a8-4ccc-ad9b-3cc648f85802/istio-proxy/0.log" Nov 03 20:58:36.143445 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:58:36.143403 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:58:39.104195 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:39.104152 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-jz4d7_428ef9c3-963d-4e46-b3f4-cf5fcc6cf0cb/manager/0.log" Nov 03 20:58:39.530286 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:39.530193 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-mq8tg_2e100b28-7be2-4dcf-abbc-3532f79ff178/server/0.log" Nov 03 20:58:39.943770 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:39.943737 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cbnpv_4c6a37ee-a6e8-4345-990b-82ef4d3d3a77/proxy/0.log" Nov 03 20:58:40.651822 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:40.651787 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78bbd54c6f-qsnrw_2857ffa5-3e45-40b8-a302-86f5083b930c/maas-api/0.log" Nov 03 20:58:41.785265 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:41.785235 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-g6q2z_704584c9-f1c3-4968-bffe-92125ac2a96c/postgres/0.log" Nov 03 20:58:43.186822 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:43.186790 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5gww6_a6e7ed4a-ecf6-4a3a-8037-e10cba476618/manager/0.log" Nov 03 20:58:43.752215 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:43.752182 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6wqqh_392af69d-c63a-4927-980c-5664b3321224/manager/0.log" Nov 03 20:58:44.136978 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:44.136944 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-jth7z_603fa1ee-3000-485f-92ad-66a3f1bd726c/manager/0.log" Nov 03 20:58:44.741078 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:44.741047 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-qrjl6_82519fb3-ab4a-42bf-a8e0-c911224c33ab/istio-proxy/0.log" Nov 03 20:58:45.693544 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:45.693508 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-dgjgp_8f7c5959-df8b-4348-8c8c-c61ba06f66d4/istio-proxy/0.log" Nov 03 20:58:45.879596 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:45.879569 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-g6znk_ad84ee89-c964-4229-be80-bb309afb1255/istio-proxy/0.log" Nov 03 20:58:46.253738 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:46.253703 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-n42qd_2e3b005c-f816-4fbd-8812-b007b7fce9e7/istio-proxy/0.log" Nov 03 20:58:46.673850 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:46.673817 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-p7lrd_93f7f2c5-96a8-4ccc-ad9b-3cc648f85802/istio-proxy/1.log" Nov 03 20:58:47.429555 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:58:47.429507 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59_80d281fa-6f07-44cf-a71f-2de479e23415/istio-proxy/0.log" Nov 03 20:59:06.149721 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:59:06.149669 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 20:59:08.883277 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:59:08.883233 4457 scope.go:122] "RemoveContainer" containerID="bc28fd22894f6a5c0f2e81e0da5488c687cf431f96651b4aa4bd7a64aec852ad" Nov 03 20:59:08.892215 ip-10-0-132-25 kubenswrapper[4457]: I1103 20:59:08.892195 4457 scope.go:122] "RemoveContainer" containerID="7ffbb0852d85ad38be29508ac377d072bd10b3ef427dabd476890eb312f65216" Nov 03 20:59:36.145459 ip-10-0-132-25 kubenswrapper[4457]: E1103 20:59:36.145358 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 21:00:06.145671 ip-10-0-132-25 kubenswrapper[4457]: E1103 21:00:06.145614 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 21:00:14.440178 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:14.440128 4457 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal" podUID="ca7a77d780aa380ba43b79db4d32277a" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 21:00:34.553774 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:34.553745 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l5wn9_e6338d09-ef6a-4cf5-8a06-bd466bc5076d/global-pull-secret-syncer/0.log" Nov 03 21:00:34.858101 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:34.858023 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9ktt5_642aaac2-4dc2-4800-bc55-c70730884400/konnectivity-agent/0.log" Nov 03 21:00:34.930495 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:34.930463 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-25.us-east-2.compute.internal_ca7a77d780aa380ba43b79db4d32277a/haproxy/0.log" Nov 03 21:00:36.144716 ip-10-0-132-25 kubenswrapper[4457]: E1103 21:00:36.144682 4457 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6tmzf" containerName="extractor" Nov 03 21:00:43.871484 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:43.871446 4457 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn"] Nov 03 21:00:43.875038 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:43.875016 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:43.880860 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:43.880835 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpp9v\"/\"kube-root-ca.crt\"" Nov 03 21:00:43.881195 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:43.881167 4457 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zpp9v\"/\"openshift-service-ca.crt\"" Nov 03 21:00:43.882608 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:43.882587 4457 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zpp9v\"/\"default-dockercfg-dsb8v\"" Nov 03 21:00:43.885581 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:43.885559 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn"] Nov 03 21:00:43.893769 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:43.893752 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-5gww6_a6e7ed4a-ecf6-4a3a-8037-e10cba476618/manager/0.log" Nov 03 21:00:43.958932 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:43.958901 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz6cf\" (UniqueName: \"kubernetes.io/projected/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-kube-api-access-sz6cf\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:43.959093 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:43.958947 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-lib-modules\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:43.959093 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:43.959049 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-proc\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:43.959200 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:43.959093 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-sys\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:43.959200 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:43.959180 4457 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-podres\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:44.013634 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.013608 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6wqqh_392af69d-c63a-4927-980c-5664b3321224/manager/0.log" Nov 03 21:00:44.059884 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.059857 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-podres\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:44.060035 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.059902 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz6cf\" (UniqueName: \"kubernetes.io/projected/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-kube-api-access-sz6cf\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:44.060035 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.059934 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-lib-modules\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:44.060035 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.059985 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-proc\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:44.060035 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.059987 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-podres\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:44.060262 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.060042 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-proc\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:44.060262 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.060045 4457 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-sys\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:44.060262 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.060082 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-lib-modules\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:44.060262 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.060089 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-sys\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:44.072125 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.072103 4457 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz6cf\" (UniqueName: \"kubernetes.io/projected/9d226d9c-6dfa-4c41-b242-b86ea31f9fe4-kube-api-access-sz6cf\") pod \"perf-node-gather-daemonset-bcdfn\" (UID: \"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4\") " pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:44.123774 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.123704 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-jth7z_603fa1ee-3000-485f-92ad-66a3f1bd726c/manager/0.log" Nov 03 21:00:44.189044 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.189020 4457 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:44.466740 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.466713 4457 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn"] Nov 03 21:00:44.468421 ip-10-0-132-25 kubenswrapper[4457]: W1103 21:00:44.468369 4457 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9d226d9c_6dfa_4c41_b242_b86ea31f9fe4.slice/crio-bf5a935aa6ef1d5a1c443f5f85d55208be2e7707343b82c0e6613123e80fcce2 WatchSource:0}: Error finding container bf5a935aa6ef1d5a1c443f5f85d55208be2e7707343b82c0e6613123e80fcce2: Status 404 returned error can't find the container with id bf5a935aa6ef1d5a1c443f5f85d55208be2e7707343b82c0e6613123e80fcce2 Nov 03 21:00:44.831631 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.831596 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" event={"ID":"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4","Type":"ContainerStarted","Data":"937b3a49541d5c026e28267fa41b1599483cc565b9877ad9144b9fe05f25cba7"} Nov 03 21:00:44.831854 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.831656 4457 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" event={"ID":"9d226d9c-6dfa-4c41-b242-b86ea31f9fe4","Type":"ContainerStarted","Data":"bf5a935aa6ef1d5a1c443f5f85d55208be2e7707343b82c0e6613123e80fcce2"} Nov 03 21:00:44.831854 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.831761 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:44.854109 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:44.854058 4457 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" podStartSLOduration=1.854045406 podStartE2EDuration="1.854045406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 21:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 21:00:44.853657343 +0000 UTC m=+3823.229254089" watchObservedRunningTime="2026-11-03 21:00:44.854045406 +0000 UTC m=+3823.229642157" Nov 03 21:00:45.520087 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:45.520056 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-62hrj_b09564c7-0bed-4d99-8cec-39a50dd69999/node-exporter/0.log" Nov 03 21:00:45.545335 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:45.545308 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-62hrj_b09564c7-0bed-4d99-8cec-39a50dd69999/kube-rbac-proxy/0.log" Nov 03 21:00:45.572131 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:45.572107 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-62hrj_b09564c7-0bed-4d99-8cec-39a50dd69999/init-textfile/0.log" Nov 03 21:00:46.124242 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:46.124216 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-rwwdh_dac45c98-1ae5-4137-bc74-943c81ac477b/prometheus-operator-admission-webhook/0.log" Nov 03 21:00:46.245937 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:46.245905 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb8966dcb-xfqv2_aafb6538-86d0-4d51-9afe-c4c49e749d1b/thanos-query/0.log" Nov 03 21:00:46.272322 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:46.272299 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb8966dcb-xfqv2_aafb6538-86d0-4d51-9afe-c4c49e749d1b/kube-rbac-proxy-web/0.log" Nov 03 21:00:46.301333 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:46.301306 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb8966dcb-xfqv2_aafb6538-86d0-4d51-9afe-c4c49e749d1b/kube-rbac-proxy/0.log" Nov 03 21:00:46.328112 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:46.328087 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb8966dcb-xfqv2_aafb6538-86d0-4d51-9afe-c4c49e749d1b/prom-label-proxy/0.log" Nov 03 21:00:46.354812 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:46.354795 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb8966dcb-xfqv2_aafb6538-86d0-4d51-9afe-c4c49e749d1b/kube-rbac-proxy-rules/0.log" Nov 03 21:00:46.393542 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:46.393471 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7bb8966dcb-xfqv2_aafb6538-86d0-4d51-9afe-c4c49e749d1b/kube-rbac-proxy-metrics/0.log" Nov 03 21:00:48.800923 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:48.800888 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-nbjzs_865485da-2a04-435a-8b33-18beb925bf8e/download-server/0.log" Nov 03 21:00:50.561931 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:50.561900 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5zxlx_731a10e2-8c1d-48c4-a453-1d1d589f6ae2/dns/0.log" Nov 03 21:00:50.628277 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:50.628249 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5zxlx_731a10e2-8c1d-48c4-a453-1d1d589f6ae2/kube-rbac-proxy/0.log" Nov 03 21:00:50.847241 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:50.847167 4457 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zpp9v/perf-node-gather-daemonset-bcdfn" Nov 03 21:00:50.919322 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:50.919292 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ssj5l_7f215e51-f5cf-4103-9904-693d5721e647/dns-node-resolver/0.log" Nov 03 21:00:51.582363 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:51.582335 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67895ffd5b-vwr6d_655da7d5-abe0-4c81-98a0-c4e2bf21b0a2/registry/0.log" Nov 03 21:00:51.634218 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:51.634191 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cwtc9_ffa0c959-6bd1-4feb-b713-3545a6896409/node-ca/0.log" Nov 03 21:00:52.937006 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:52.936974 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-qrjl6_82519fb3-ab4a-42bf-a8e0-c911224c33ab/istio-proxy/0.log" Nov 03 21:00:53.725533 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:53.725495 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-dgjgp_8f7c5959-df8b-4348-8c8c-c61ba06f66d4/istio-proxy/0.log" Nov 03 21:00:53.757271 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:53.757243 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-g6znk_ad84ee89-c964-4229-be80-bb309afb1255/istio-proxy/0.log" Nov 03 21:00:53.837319 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:53.837285 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-n42qd_2e3b005c-f816-4fbd-8812-b007b7fce9e7/istio-proxy/0.log" Nov 03 21:00:53.913478 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:53.913445 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-p7lrd_93f7f2c5-96a8-4ccc-ad9b-3cc648f85802/istio-proxy/0.log" Nov 03 21:00:53.914494 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:53.914478 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-p7lrd_93f7f2c5-96a8-4ccc-ad9b-3cc648f85802/istio-proxy/1.log" Nov 03 21:00:54.072668 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:54.072632 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59_80d281fa-6f07-44cf-a71f-2de479e23415/istio-proxy/0.log" Nov 03 21:00:55.412440 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:55.412410 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-qrjl6_82519fb3-ab4a-42bf-a8e0-c911224c33ab/istio-proxy/0.log" Nov 03 21:00:55.648152 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:55.648122 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-dgjgp_8f7c5959-df8b-4348-8c8c-c61ba06f66d4/istio-proxy/0.log" Nov 03 21:00:55.665810 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:55.665730 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-g6znk_ad84ee89-c964-4229-be80-bb309afb1255/istio-proxy/0.log" Nov 03 21:00:55.704998 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:55.704969 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-n42qd_2e3b005c-f816-4fbd-8812-b007b7fce9e7/istio-proxy/0.log" Nov 03 21:00:55.750798 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:55.750770 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-p7lrd_93f7f2c5-96a8-4ccc-ad9b-3cc648f85802/istio-proxy/0.log" Nov 03 21:00:55.752030 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:55.752010 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-p7lrd_93f7f2c5-96a8-4ccc-ad9b-3cc648f85802/istio-proxy/1.log" Nov 03 21:00:55.829432 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:55.829402 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59_80d281fa-6f07-44cf-a71f-2de479e23415/istio-proxy/0.log" Nov 03 21:00:56.546957 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:56.546926 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qxxzr_ac15ca75-0dd3-4f05-b21d-1ae2280d0415/serve-healthcheck-canary/0.log" Nov 03 21:00:57.119094 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:57.119066 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6tmzf_224062e4-7b08-4d73-ba03-1789a9f591be/kube-rbac-proxy/0.log" Nov 03 21:00:57.156532 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:57.156507 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6tmzf_224062e4-7b08-4d73-ba03-1789a9f591be/exporter/0.log" Nov 03 21:00:57.183586 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:57.183562 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6tmzf_224062e4-7b08-4d73-ba03-1789a9f591be/extractor/0.log" Nov 03 21:00:58.925063 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:58.925033 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-57dc67465-qrjl6_82519fb3-ab4a-42bf-a8e0-c911224c33ab/istio-proxy/0.log" Nov 03 21:00:59.140792 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:59.140758 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-dgjgp_8f7c5959-df8b-4348-8c8c-c61ba06f66d4/istio-proxy/0.log" Nov 03 21:00:59.157757 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:59.157723 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-g6znk_ad84ee89-c964-4229-be80-bb309afb1255/istio-proxy/0.log" Nov 03 21:00:59.193097 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:59.193030 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-n42qd_2e3b005c-f816-4fbd-8812-b007b7fce9e7/istio-proxy/0.log" Nov 03 21:00:59.229439 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:59.229411 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-p7lrd_93f7f2c5-96a8-4ccc-ad9b-3cc648f85802/istio-proxy/0.log" Nov 03 21:00:59.230512 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:59.230487 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-p7lrd_93f7f2c5-96a8-4ccc-ad9b-3cc648f85802/istio-proxy/1.log" Nov 03 21:00:59.387321 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:00:59.387294 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-689986b98c-zjj59_80d281fa-6f07-44cf-a71f-2de479e23415/istio-proxy/0.log" Nov 03 21:01:00.050496 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:01:00.050461 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-79746d584b-zn6wd_81c8482e-e941-49ed-ad35-73866f7e6bed/manager/0.log" Nov 03 21:01:00.080106 ip-10-0-132-25 kubenswrapper[4457]: I1103 21:01:00.080082 4457 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-79746d584b-zn6wd_81c8482e-e941-49ed-ad35-73866f7e6bed/copy-manifests/0.log"